Certification Frequently Asked Questions
V1.3a
Dec 5 2012
General Questions
Q.
What is the POSIX®: Certified by IEEE and The Open Group Certification Program?
Q.
Q.
Q.
Q.
What is the difference between Platform Specific and Product Family certification?
Q.
What test suites do I need and where can they be obtained from?
Q.
Q.
Q.
Q.
Q.
How many tests are involved in POSIX®: Certified by IEEE and The Open Group Certification?
Q.
Q.
Q.
Q.
Who do we contact regarding the Trademark License Agreement (TMLA)?
Q.
Can the Trademark be licensed for use with all Product Standards?
Q.
Q.
Q.
Q.
Q.
Issues during Certification
Q.
Q.
When installing VSX-PCTS2003 which subsets should i select when running config.sh?
Q.
Does VSX-PCTS2003 support cross compilation for a remote target?
Q.
The product fails one or more of the tests in the test campaign. What happens now?
Q.
Q.
We are using the test suites and some of the tests are failing or hanging. Can you help?
Q.
Q.
Problem Reporting
Q.
Q.
Q.
Where can I find out what has happened to the Problem Report I sent in a week ago?
General Questions
What is the POSIX®: Certified by IEEE and The Open Group Certification Program?
The POSIX®: Certified by IEEE and The Open Group Certification program is a voluntary program of the IEEE and The Open Group, open to any product meeting the conformance requirements.
More information on why this is a joint program (see Q3)
The key documents describing the obligations, terms and conditions of POSIX®: Certified by IEEE and The Open Group certification are:
the POSIX Certification Policythe Certification Agreement
the POSIX® Trademark License Agreement
the Certification Guide
The technical requirements for a conforming system are described in detail in the Product Standards.
What are the advantages of Certification?
POSIX®: Certified by IEEE and The Open Group certification is widely supported and accepted by government and other major procurers of IT devices. To compete with the major suppliers whose products are certified you may wish to consult your intended customers to ascertain their needs. We believe the return on investment is substantial.
Which documents should I read first to become familiar with the POSIX®: Certified by IEEE and The Open Group Certification Program?
To become familiar with the POSIX®: Certified by IEEE and The Open Group Certification program, you should read the following program documents:
The Certification Guide
This document should be read first. It gives the overview of the program,
what it means to be certified,
what is required to get a product certified and how to make sure that a
product remains certified.
The Certification Agreement
The Certification Agreement is between you and the Certification Authority. It defines the certification service and the legal commitment to the conditions of the service. It applies to each product being certified.
The POSIX® Trademark License Agreement
The POSIX® Trademark License Agreement is between you and the IEEE and
requires signature. It is recommended that you commence the process to complete the Trademark License Agreement as soon as possible
to save delays later in the certification process. A trademark license is optional but mandatory for use of the POSIX trademark, it must be completed before a product can complete the
certification process and use the mark. Only certain classes of
certification qualify to license the mark (see the Guide for details).
The Trademark License agreement only needs to be entered into once per organization. When it is in place multiple products can be certified.
The Product Standards
The Product Standards should then be read to understand the detailed conformance requirements against which a product
can be certified. Product Standards provide a mapping between certification, the specifications required and the test suites needed to demonstrate conformance.
Where can I obtain the POSIX standards from?
The html version of the latest version of the POSIX 1003.1 standard is freely available to read and download from links at: URL:http://www.unix.org/, you need to register for a copy.
PDF copies of the standard can also be obtained either from The IEEE (search on 1003.1) , The Open Group (Look for documents T041,C082) or ISO (look for ISO/IEC 9945 ).
For other POSIX standards see the IEEE Standards web site
What is the difference between Platform Specific and Product Family certification?
This is the Level of Certification as defined in the Certification Policy.
Platform Specific Certification applies to a single defined environment (i.e. a single hardware and specific operating software environment combination), whereas Product Family Certification applies to all members of a binary-compatible family (i.e. multiple hardware environments running the same binary compatible operating system).
For testing a single test report is required from the platform being submitted for Platform Specific certification, whereas test reports from two members of the binary compatible are required for a Product Family certification.
What test suites do I need and where can they be obtained from?
The list of test suites currently authorized for use in the POSIX®: Certified by IEEE and The Open Group Certification program, is available here. You can license some of the test suites (for 1003.1 Base) for no charge (a 90 day license) or take out a commercial license (available for longer durations). The 90 day licenses should be downloaded from here, signed and returned to The Open Group. Other test suites are only available under a commercial license. The exact test suites required depends on the product standard and in some instances the test campaign selected, information is available here.
How do I get help running the test suites?
For the 90 day free licenses no support is available other than through the formal Problem Reporting system.
The Open Group also offers commercial support contracts for all its test suites referenced in the program, which includes email support provided through the web at http://www.opengroup.org/testing/support/ . This is recommended if you have no experience with the test suites and are undergoing product development. It can reduce your overall effort to develop a conforming product.
I want to certify a product which is based on a model previously certified. Is the procedure the same even though the software hasn't changed? Does certification cover the same product recompiled for different processor architectures?
The Certification Policy allows for an existing certified product to be renamed without the need for further certification or testing. You will be required to provide a written statement to the CA indicating that there have been no material changes to the certified product. If there are changes then it depends on the nature of the changes, please refer to the TMLA.
The same product on a different processor architecture, even if built from the same source constitutes a new product with respect to the certification requirements, and is subject to a full test and certification. The act of recompilation is a material change and requires demonstration of conformance.
What is required of the software version that the certification is started with? Does it have to be commercially available, or is a R&D version sufficient? How does the version need to be identified?
Certification may take place at any stage of the product's life cycle. If under development you may wish to certify the product prior to formal product launch, with the certification being held confidential prior to a public launch. The product declared on the public register needs to be commercially available for procurement. The software version of the product should be clearly identifiable. If the R&D name and version is very different from the commercial release then we have to be assured that it is indeed a maintenance release. Please communicate with the Certification Authority regarding any uncertainties. We will respond to you regarding the scope of testing if, for any reason, the situation is not clear.
How long does it take to test and become certified?
Timescales and preparations for formal testing depend on each situation, so it is not possible for us to provide an estimate. Timescales also depend on the Product Standard, since some are more complex than others.
For those testing with test suites under the 90 day time-limited free license, The Open Group will allow one extension of 90 days, so the initial test period could be up to 180 days. If during submission of the results package and supporting document, a resubmission is required and you are within 60 days of the end of the time-limited license an extension of 60 days will be granted, so potentially the limit is 239 days.
From the time we receive your complete submission,including conformance statement and test results, the normal timeframe is up to 6 working days for certificaton to be confirmed. This assumes there are no queries or issues.If issues are found, then you have up to 60 days to correct and resubmit. Just one resubmission is included in the certification fee.
How many tests are involved in POSIX®: Certified by IEEE and The Open Group Certification?
This depends on the Product Standard: For 1003.1-2003 System Interfaces there are approximately 17,800 tests. For 1003.1-2003 Shell and Utilities there are approximately 5,000 tests.
How long does it take to do test runs?
Again this depends on the Product Standard, the speed of the machine under test and options supported. Typical execution times for VSX-PCTS2003 and VSC-PCTS2003 are each 12 hours to execute.
How long does Certification last ?
Once you have tested and your results have been approved as conforming to the current POSIX®: Certified by IEEE and The Open Group Specification, you will be certified for a period of twelve months. If you are given permission to OEM or Re-Badge another companies product then your product will have the same renewal date as the original certified product on which it is based.
Who do we contact for Certification information?
POSIX®: Certified by IEEE and The Open Group Certification Authority
The Open Group
Apex Plaza, Forbury Road
Reading, Berks RG1 1AX
United Kingdom
Tel: +44 0118 950 8311
Fax: +1 415 276 3760
Email: POSIX®: Certified by IEEE and The Open Group Certification Authory
Who do we contact regarding the Trademark License Agreement (TMLA)?
Any questions about the TMLA should be addressed to: the IEEE as detailed in the the trademark license agreement.
Can the Trademark be licensed for use with all Product Standards?
The POSIX® trademark can be licensed for use only with the 1003.1-2003 Base Product Standard, the PSE54 Multipurpose Realtime Product Standard and the PSE52 Multipurpose Realtime Product Standard, denoted by the presence of the Label for Logo section of the Product Standard.
Note it can not be licensed for the 1003.1-2003 System Interfaces or 1003.1-2003 Shell and Utilities separately. The rationale behind this is that the POSIX 1003.1 standard defines its POSIX Conformance requirements in Section 2 of the standard to include conformance to both System Interfaces and Shell and Utilities.
What can I say about my product once its certified?
This depends on two factors: the Product Standard that the product has been certified against, and the Level of certification (platform specific versus product family).
-
Product Standard :
- For Product Standards that do not have the Label for Logo section defined, the POSIX trademark cannot be licensed nor the certification logos used. For this class of Product Standard, once the CA has notified a Supplier that a Product is a Certified Product, the Supplier may make a statement that the Product is certified to the (Insert Product Standard Name) and is listed on the certification register at http://posix.opengroup.org/register.html .
- For Product Standards where a Label for Logo section is defined, for example the 1003.1-2003 Base Product Standard, if the POSIX trademark has been licensed, then the trademark usage guidelines should be consulted. Otherwise the statement is as above. Note that certification in itself confers no rights to use the POSIX trademark.
-
Level of Certification :
- Platform Specific certification is for a single platform instance and so statements about the certification can only be in connection with the specific combination of the Software and Hardware environment that has been registered.
- Product Family certification is for a binary compatible family of operating system and hardware environments, and thus statements about the certification can apply to all members of the range of the binary compatible family that are defined in the conformance statement.
Do I get a certificate?
If you have successfully certified a product to a Product Standard for which the POSIX trademark can be licensed, and are a POSIX trademark licensee you will be issued a certificate.
How much does certification cost?
Please see the fee schedule by clicking here.
How can I pay the certification fee?
Payment can be made by credit card. Alternatively, payment may be made by check or wire transfer. Note that payment methods other than by credit card can delay certification until the payment is received.
When do I have to pay the certification fee?
The fee is due when you complete submission of the product registration. The Certification Authority will process your registration after the fee is paid.
Issues during Certification
How can we get the Conformance Statements?
To view the Conformance Statements please go to http://posix.opengroup.org/docs/conformance.html
When installing VSX-PCTS2003 which subsets should i select when running config.sh?
During the configuration stage (config.sh), the script will output the following:
Which subsets do you wish to install? Enter a space-separated list including one or more of: base base6 lfs mse [base base6 lfs mse]
You should select all four, i.e. base, base6, lfs and mse.
Note: The default list offered in the square brackets varies depending on previous configurations. So pressing return at this point might not always select all the subsets. The first time the default would be the full list. But if only some of the subsets are selected this time, then those will be the ones offered as the default next time.
Does VSX-PCTS2003 support cross compilation for a remote target?
Yes, support for cross-compilation is provided in the version of the Generic VSXgen framework provided as part of VSX-PCTS2003. If at at configuration you select "yes" to "Do the test executables need to be cross-compiled?", then when you build the tests, the build script will set the various compiler configuration paramaters for the cross-compiler values:
CC=$XCC COPTS=$XCOPTS LDFLAGS=$XLDFLAGS SYSLIBS=$XSYSLIBS AR=$XAR RANLIB=$XRANLIB
Further instructions on how to run the VSX-PCTS2003 in a cross compiled environment are available on request.
The product fails one or more of the tests in the test campaign. What happens now?
Please check the Problem Report Database first. If your fail results are covered by agreed test suite deficiencies (TSD's) for the version of the test suite, or agreed interpretations of the specificaton, then certification may be able to proceed. If you have submitted your test results for certification and you have unresolved fail results, your certification fee covers two attempts to certify.
We have encountered some problems with the POSIX®: Certified by IEEE and The Open Group Test Suites. What do we do?
If you believe there is a bug in the test suite, or the Certification system, or a specification, please raise a Problem Report giving enough detail for the problem to be reproduced. This will enable any fault with the test suite and the particular version to be dealt with satisfactorily. If you are having difficulty running tests or operating the test suite, and you have a support contract please report the problem to the Test Suite Maintenance Authority (see http://www.opengroup.org/testing/support/)
We are using the test suites and some of the tests are failing or hanging. Can you help?
The Certification Authority does not provide support for the test suite and is unable to provide help directly in debugging products. If you have a support contract, please contact the Test Suite Maintenance Authority (as noted above), please provide sufficient information for then to determine what the problem is ( for example, specific syntax or logic error in the content and the journal output).
We started certification testing for a product and while testing was taking place, we found there are product problems requiring some bug-fixing modifications. Do we have to rerun the entire tests in the campaign?
For certification we need clear runs of the test suite. If your product requires bug fixing during the test cycle, the fixes should be made and the product re-tested from the beginning if you want to certify the latest release of your product. After certification, if more product changes are required, the requirements of the Certification Agreement, Certification Policy and the TMLA (if applicable) apply.
I have some FIP results in the automated tests. How Do I handle these?
For FIP results (further information provided) you need to sign off within the report that the manual resolution of the test is a pass result.
An explanation of the result codes from the automated test suites which use TET follows below:
PASS - a test result belonging to this group is considered to be a pass for certification purposes and does not need reference to any granted interpretations or waivers:
Pass - the test has been executed correctly and to completion without any kind of problem
Warning - the functionality is acceptable, but you should be aware that later revisions of the relevant standards or specification may change the requirements in this area.
FIP - additional information is provided which needs to be checked manually.
Unsupported - an optional feature is not available or not supported in the implementation under test.
Not in Use - some tests may not be required in certain test modes or when an interface can be implemented by a macro or function and there are two versions of the test only one is used.
Untested - no test written to check a particular feature or an optional facility needed to perform a test is not available on the system.
FAIL - a test result belonging to this group is considered to be a fail for certification purposes (unless the failure has been waived by an agreed interpretation or waiver in the Interpretations database):
Fail - the interface did not behave as expected.
Uninitiated - the particular test in question did not start to execute.
Unresolved - the test started but did not reach the point where the test was able to report success or failure.
Unreported - a major error occurred during the testset execution.
Problem Reporting
Where do I report a bug in the specification/test suite?
If the bug is preventing you from certifying a product it should be reported into the POSIX®: Certified by IEEE and The Open Group Certification Authority Problem Reporting system . From the Problem Reporting entry page select "Search/Submit", and then on the next screen "Submit an Interpretation (right hand column)". You are then presented with a screen where you must select the name of the Product Standard -- please select the lowest level product standard.
If the bug does not concern certification, it should be reported into the test suite support team directly at http://www.opengroup.org/testing/support/
What must I check before referencing a Problem Report?
Please check that the status of the Problem Report is either a test suite deficiency (TSD), certification system deficiency (CSD) or an interpretation (INT). REJ means it was not agreed and therefore the PR may not be referenced. Please also check that the PR applies to the certification release of the test suite, which you are using. Please note that TSDs are test suite specific.
Where can I find out what has happened to the Problem Report I sent in a week ago?
You can track problem report status on the Problem Report web site.