taucmdr.cf.compiler package

TAU compiler knowledgebase.

TAU Commander uses this knowledgebase to try to work out what kind of compiler the user is using to build their code. We can get away with this because TAU will take care of most of the version-specific details if only we get the configuration line correct.

TAU depends very strongly on compiler characteristics. Each TAU configuration is only valid for a single compiler, TAU’s feature set changes depending on compiler family and version, TAU’s configure script is easily confused by compilers with strange names or unusual installation paths, etc. In most cases, TAU doesn’t even try to detect the compiler and trusts the user to specify the right “magic words” at configuration. Worse, TAU sometimes does probe the compiler to discover things like MPI headers, except it does such a poor job that many time we get the wrong answer or simply cause the configure script to fail entirely. This has been a major painpoint for TAU users.

The reason for this mess is that compiler detection is very hard. Projects like SciPy that depend on a compiler’s stdout stream for compiler detection have more or less failed in this since compiler messages are exceptionally difficult to parse. CMake does a good job by invoking the compiler command and parsing strings out of a compiled object to detect compiler characteristics, but that approach is complex and still breaks easily.

The TAU Commander compiler knowledgebase associates a compiler command (icc) with characteristics like family (Intel) and role (CC). Any compiler commands intercepted by TAU Commander are matched against the database to discover their characteristics.

The knowledgebase lists all compilers known to TAU Commander but only some of those will actually be available on any given system. Use InstalledCompiler and related classes to discover installed compilers and to determine features that change from system to system.

class taucmdr.cf.compiler.InstalledCompiler(absolute_path, info, **kwargs)

Bases: object

Information about an installed compiler command.

There are relatively few well known compilers, but a potentially infinite number of commands that can invoke those compilers. Additionally, an installed compiler command may be a wrapper around another command. This class links a command (e.g. icc, gcc-4.2, etc.) with a compiler command in the knowledgebase.

absolute_path

Absolute path to the compiler command.

Type:

str

info

Information about the compiler invoked by the compiler command.

Type:

_CompilerInfo

command

Command that invokes the compiler, without path.

Type:

str

path

Absolute path to folder containing the compiler command.

Type:

str

uid

A unique identifier for the installed compiler.

Type:

str

wrapped

Information about the wrapped compiler, if any.

Type:

InstalledCompiler

include_path

Paths to search for include files when compiling with the wrapped compiler.

Type:

list

library_path

Paths to search for libraries when linking with the wrapped compiler.

Type:

list

compiler_flags

Additional flags used when compiling with the wrapped compiler.

Type:

list

libraries

Additional libraries to link when linking with the wrapped compiler.

Type:

list

_parse_wrapped_args(args)

_probe_wrapper()

classmethod find_any(role)

Find any installed compiler that can fill the given role.

Iterates through all known compiler families and probes for an installed compiler matching the specified role.

Parameters:

role (_CompilerRole) – The compiler role to fill (e.g. CC, CXX, FC).

Raises:

ConfigurationError – No installed compiler found for the role.

generate_wrapper(prefix)

Generate a compiler wrapper script that uses taucmdr.TAUCMDR_SCRIPT to invoke the compiler.

Parameters:

prefix (str) – Path to a directory in which the wrapper script will be created.

classmethod probe(command, family=None, role=None)

Probe the system to discover information about an installed compiler.

Parameters:

• command (str) – Absolute or relative path to an installed compiler command.

• family (_CompilerFamily) – Installed compiler’s family if known, None otherwise.

• role (_CompilerRole) – Installed compiler’s role if known, None otherwise.

Raises:

ConfigurationError – Unknown compiler command or not enough information given to perform the probe.

Returns:

A new InstalledCompiler instance describing the compiler.

Return type:

InstalledCompiler

unwrap()

Iterate through layers of compiler wrappers to find the true compiler.

Returns:

Compiler wrapped by this compiler, self if this compiler doesn’t wrap another.

Return type:

InstalledCompiler

property version

Get the compiler’s self-reported version info.

Usually whatever the compiler prints when the –version flag is provided.

Returns:

The compilers’ version (usually) in the format (maj, min, build).

Return type:

tuple

property version_string

Get the compiler’s self-reported version info.

Usually whatever the compiler prints when the –version flag is provided.

Returns:

The compilers’ version string.

Return type:

str

class taucmdr.cf.compiler.InstalledCompilerCreator

Bases: type

Cache compiler probe results.

InstalledCompiler invokes a compiler command to discover system-specific compiler characteristics. This can be very expensive, so this metaclass changes the instance creation procedure to only probe the compiler when the compiler command has never been seen before and avoid duplicate invocations in a case like:

a = InstalledCompiler('/path/to/icc')
b = InstalledCompiler('/path/to/icc')

Without this metaclass, a and b would be different instances assigned to the same compiler and icc would be probed twice. With this metaclass, b is a == True and icc is only invoked once.

class taucmdr.cf.compiler.InstalledCompilerFamily(family)

Bases: object

Information about an installed compiler family.

Compiler families are usually installed at a common prefix but there is no guarantee that all members of the family will be installed. For example, it is often the case that C and C++ compilers are installed but no Fortran compiler is installed. This class tracks which members of a compiler family are actually installed on the system.

family

The installed family.

Type:

_CompilerFamily

FIXME

members

__contains__(role)

Returns True if any installed compiler can fill a given role.

__getitem__(role)

Return the preferred installed compiler for a given role.

Since compiler can perform multiple roles we often have many commands that could fit a given role, but only one preferred command for the role. For example, icpc can fill the CC or CXX roles but icc is preferred over icpc for the CC role.

Parameters:

role (_CompilerRole) – The compiler role to fill.

Returns:

The installed compiler for the role.

Return type:

InstalledCompiler

Raises:

• KeyError – This family has no compiler in the given role.

• IndexError – No installed compiler fills the given role.

__iter__()

Yield one InstalledCompiler for each role filled by any compiler in this installation.

get(role, default=None)

class taucmdr.cf.compiler.InstalledCompilerSet(*args, **kwargs)

Bases: KeyedRecord

A collection of installed compilers, one per role if the role can be filled.

To actually build a software package (or user’s application) we must have exactly one compiler in each required compiler role.

uid

A unique identifier for this particular combination of compilers.

members

(_CompilerRole, InstalledCompiler) dictionary containing members of this set

_add_members(**kwargs)

items()

keys()

Get the name of the key field in all instances.

Returns:

All instance keys.

Return type:

list

modify(**kwargs)

Build a modified copy of this object.

values()

class taucmdr.cf.compiler.Knowledgebase(keyword, description, **kwargs)

Bases: object

TAU compiler knowledgebase front-end.

add(name, *args, **kwargs)

Add a new compiler family to the knowledgebase.

Compilers in the family are specified via keyword arguments. The key is the same that was used to define the role in the knowledgebase not _CompilerRole.keyword, which is calculated from the knowledgebase name and the role keyword. The value is a string or collection of strings specifying compiler commands, e.g. ‘gcc’ or (‘xlf’, ‘xlf_r’).

Parameters:

• name (str) – The compiler family name.

• *args – Positional arguments passed to the _CompilerFamily constructor.

• **kwargs – Keyword arguments passed to the _CompilerFamily constructor, after any keyword arguments specifying compiler commands are removed.

Returns:

The new compiler family object.

Return type:

_CompilerFamily

classmethod all_compilers()

classmethod all_roles()

Return all known compiler roles.

property families

family_names()

Return an alphabetical list of all compiler family names.

classmethod find_compiler(command=None, family=None, role=None)

classmethod find_role(keyword)

Find a compiler role with the given keyword.

iterfamilies()

Iterate over compiler families in the knowledgebase.

The first family yielded is the host’s preferred compiler family. Fallback families are yielded last; all other families may be yielded in any order.

property roles

class taucmdr.cf.compiler._CompilerFamily(*_args, **_kwargs)

Bases: TrackedInstance

Information about a compiler family.

A compiler’s family creates associations between different compiler commands and assigns compilers to roles. All compiler commands within a family accept similar arguments, and produce compatible object files.

kbase

The knowledgebase instance that created this object.

Type:

Knowledgebase

name

Family name, e.g. “Intel”.

Type:

str

family_regex

Regular expression identifying compiler family in compiler version string.

Type:

str

version_flags

Command line flags that show the compiler version, e.g. ‘–version’.

Type:

list

include_path_flags

Command line flags that add a directory to the compiler’s include path, e.g. ‘-I’.

Type:

list

library_path_flags

Command line flags that add a directory to the compiler’s library path, e.g. ‘-L’.

Type:

list

link_library_flags

Command line flags that link a library, e.g. ‘-l’.

Type:

list

show_wrapper_flags

Command line flags that show the wrapped compiler’s command line, e.g. ‘-show’.

Type:

list

optional_wrapper

If True then a command that rejects show_wrapper_flags is treated as a compiler that doesn’t wrap another compiler instead of as an error.

Type:

bool

fallback

If True then the family is searched for installed compilers only after every other family.

Type:

bool

members

Compilers in this family indexed by role.

Type:

dict

_probe_cache = {}

installation()

classmethod probe(absolute_path, candidates=None)

Determine the compiler family of a given command.

Executes the command with version_flags from all families and compares the output against family_regex.

Parameters:

• absolute_path (str) – Absolute path to a compiler command.

• candidates (list) – If present, a list of families that are most likely.

Raises:

ConfigurationError – Compiler family could not be determined.

Returns:

The compiler’s family.

Return type:

_CompilerFamily

class taucmdr.cf.compiler._CompilerInfo(*_args, **_kwargs)

Bases: TrackedInstance

Information about a compiler.

A compiler’s basic information includes it’s family (e.g. Intel) and role (e.g. CXX). The compiler might not be installed.

family

The family this compiler belongs to.

Type:

_CompilerFamily

command

Command without path or arguments, e.g. ‘icpc’

Type:

str

role

This compiler’s primary role in the family.

Type:

_CompilerRole

short_descr

A short description for command line help.

Type:

str

classmethod _find(command, family, role)

classmethod find(command=None, family=None, role=None)

Find _CompilerInfo instances that matches the given command and/or family and/or role.

Parameters:

• command (str) – A compiler command without path.

• family (_CompilerFamily) – Compiler family to search for.

• role (_CompilerRole) – Compiler role to search for.

Returns:

_CompilerInfo instances matching given compiler information.

Return type:

list

class taucmdr.cf.compiler._CompilerRole(*args, **kwargs)

Bases: KeyedRecord

Information about a compiler’s role.

A compiler role identifies how the compiler is used in the build process. All compilers play at least one role in their compiler family. Many compilers can play multiple roles, e.g. icpc can be a C++ compiler in the CXX role, a C compiler in the CC role, or even a linker. TAU Commander must identify a compiler’s role so it can configure TAU.

keyword

Name of the compiler’s role, e.g. ‘CXX’.

Type:

str

language

Name of the programming language corresponding to the compiler role, e.g. ‘C++’.

Type:

str

envars

Environment variables commonly used to identify the compiler in this role, e.g. “CXX” or “CC”.

Type:

tuple

kbase

The knowledgebase instance that created this object.

Type:

Knowledgebase

taucmdr.cf.compiler._without_minor_version(command)

Return command without a trailing .<digits>, e.g. python3.12 becomes python3.

Subpackages

• taucmdr.cf.compiler.tests package
  • Submodules
    • taucmdr.cf.compiler.tests.test_host module
      • CrayFamilyTest
      • FamilyOrderTest
      • ProbeTest
      • _probe_family()
    • taucmdr.cf.compiler.tests.test_installed module
      • GenerateWrapperTest
      • InstalledTest
    • taucmdr.cf.compiler.tests.test_mpi module
      • MpiTest
    • taucmdr.cf.compiler.tests.test_python module
      • VersionedNameTest
      • versioned_python_link()
    • taucmdr.cf.compiler.tests.test_shmem module
      • OpenSHMEMWrapperTest

Submodules

• taucmdr.cf.compiler.caf module
• taucmdr.cf.compiler.cuda module
• taucmdr.cf.compiler.host module
• taucmdr.cf.compiler.mpi module
• taucmdr.cf.compiler.python module
• taucmdr.cf.compiler.shmem module