June 2002   Vol. 3 Issue 6

 

Inside This Issue  
     


Subscribe to DDC-I Online News and receive monthly updates automatically


Previous Issues



Success Stories

Current Products

****** NEW! ******
Custom Solutions


SCORE® for OSE
A Strong Platform for Safety Critical Software Development

By Peder Møller
Software Engineer, DDC-I A/S

The integration of the SCORE® (Safety-Critical, Object-oriented, Real-time Embedded) suite of programming and testing tools with the OSE RTOS (Real-Time Operating System) from OSE Systems combines a compiler and an RTOS that are both designed for use in high-integrity embedded systems. For the first time, SCORE’s multi-language support provides the application developer the ability to utilize both Ada and C for developing OSE applications.

The combination of Ada’s reliability provided through strong typing and rigorous checks and the built-in safety features of the certified OSE RTOS, offers a strong platform for safety-critical software development. The SCORE® IDE with OSE support runs on Solaris and Windows NT platforms and can be used for generating PowerPC cross applications or for generating Solaris native applications running on the OSE SoftKernel. SCORE’s OSE support encompasses both compiler, GUI, and Multi-Language Debugger (MLD) support.

The OSE RTOS
The OSE RTOS is designed for use in fault-tolerant systems. It has been certified according to the international IEC 61508 safety standard widely used in industrial systems and is certifiable according to the DO-178B standard for avionics systems. The OSE kernel is fully preemptive with deterministic real-time behavior.

OSE achieves a high level of safety by restricting interprocess communication to a safe and flexible message passing scheme that avoids the need for shared memory. The efficient encapsulation of OSE processes makes it possible to utilize the OSE MMU support to get full memory protection between processes.

Furthermore, the OSE kernel contains extensive internal error checks and a flexible error handling mechanism. The safety guidelines in the OSE documentation helps the application developer to employ OSE for producing highly reliable applications.

OSE includes numerous embedded communication products that makes it particularly well suited for distributed systems. An OSE application can for instance be configured with OSE processes running on multiple CPUs and multiple boards communicating through TCP/IP, and the OSE signal communication can be configured so that it is transparent to the OSE processes whether other processes are local or remote.

OSE has preconfigured Board Support Packages (BSPs) for a wide range of PowerPC target boards. The OSE BSPs contain all the required board specific code for handling the board initialization and drivers for interrupt controllers, serial communication units, and ethernet devices.

SCORE® Support for OSE
SCORE® supports compilation, linking, and debugging of applications running on top of the OSE RTOS. The SCORE® OSE applications can be written in Ada or C, or in a mixture of the two languages. Both Ada and C code may use OSE processes to support multi-process applications.

The SCORE® system has a clean and board-independent interface to OSE which is provided through SCORE’s OSE UCC. This interface is the link between the SCORE® target libraries, primarily the SCORE® Run-Time System, and OSE. The interface has been designed to ensure that hardware and board specific details are entirely within the OSE domain. This means that the SCORE® target code is 100 % board independent and that SCORE® OSE applications can run on any target that has an OSE BSP.

The Ada root library has been extended with interface packages containing Ada bindings to the OSE system calls providing a seamless interface from Ada to OSE while maintaining the safety of Ada’s inherent type checking. A special variant of the OSE interface package containing only the IEC 61508 and DO-178B certified subset of the OSE system calls allows the user to check at compile-time whether the application is restricted to use this subset.

The complexity of generating an OSE executable based on the numerous configuration settings is harnessed by a few setup files utilizing the dmake tool to automatically generate OSE executables based on the configuration settings. The dmake tool is delivered with OSE and exists in command line versions on both Solaris and Windows NT. The OSE dmake setup performs all stages of the OSE application program build.

The SCORE® OSE integration has added support for compiling Ada sources and maintaining Ada libraries within the dmake framework.

Based on personal preferences the SCORE® OSE user can choose either to work from the command line using OSE’s dmake setup, or the user can choose to perform the project development entirely from within the SCORE® GUI. The SCORE® GUI support for building OSE applications is twofold:

  1. The GUI interfaces to the OSE dmake system to control the OSE setup in a given project

  2. The GUI performs compilation of the application sources, linking of the executable, and source code debugging similar to the way this is done for non-OSE applications with the only difference being a few extra link and debug options.

The SCORE® MLD can be used either from the GUI or from the command line to perform the usual Ada and C source code debugging on OSE applications. The MLD interfaces via ethernet to the OSE debug server on the PowerPC target board to provide Run-Mode debugging. The communication between the MLD and the embedded OSE debug server is completely independent of the host OS and the target board. The only requirement is that there is an ethernet connection between the host computer and the target board. The OSE Illuminator which provides process level debugging of OSE applications can be run concurrently with the MLD.

OSE executables can be linked as monoliths or load_modules. A monolith consists of the OSE kernel, the relevant embedded OSE products, and the application code all joined into one single executable which is downloaded to the target and started like any other executable. A load_module has no kernel or other embedded OSE products included. The load_module only contains the application code and it is linked as a relocatable executable which can be placed anywhere in RAM at load time. Running a load_module requires that an OSE monolith has already been downloaded to and started on the target board, and that the monolith has the OSE products included that the load_module requires. The monolith must also contain the OSE Program Handler which assists in the download of the load_module. When a downloaded load_module gets started, it dynamically hooks up with the resources it needs in the OSE monolith that is already running. With the load_module approach, OSE functions like an OS which runs all the time independent of when applications get started and terminated.

The SCORE® OSE integration supplements the OSE installation’s application examples with new examples of pure Ada and mixed Ada and C applications which allow the user a quick way to learn how these applications can be configured and how the Ada interface to OSE can be utilized.

[ Back to Top ]

On the Front Lines

Johan O. Nielsen
DDC-I A/S
Engineering Manager

As Engineering Manager for DDC-I, A/S, Johan O. Nielsen is responsible for all engineering activities in Europe and Asia. Additionally, Johan is the Assistant Product Champion for the SCORE® product line. In his role as Assistant SCORE® Product Champ, Johan works very closely with SCORE® customers as well as the SCORE® development team, deciding the best approach to each customer issue and future releases.

During his 16+ year career at DDC-I, Johan has worked at both Europe and U.S. technology centers. His expertise in the Ada programming language, multi-programming language integration and code generation particularly RISC processors, has proven to be a valuable asset. He also serves as an active evaluator and reviewer in the European Union Research Program.

Johan lives with his wife Helle and daughter Louise, in Lyngby, a suburb of Copenhagen, Denmark. He enjoys Mediterranean cooking, fine wine, travel, and has a special interest in cars, specifically the large American and fast Italian variety.

[ Back to Top ]

3rd Party Update

OSE Systems for DDC-I

Corporate Headquarters
OSE Systems, Inc.
1731 Technology Drive
San Jose, CA 95110
Toll-free (866) 844-RTOS
Tel: (408) 392-9300
Email: info@enea.com

OSE Systems is the technological provider for real-time operating systems (RTOS) and development tools for distributed and fault-tolerant applications. The OSE™ RTOS was designed from the ground up to satisfy the requirements of today’s complex embedded systems. System designers focusing on reliability, scalability, and simplicity are increasingly making OSE their platform of choice. In addition to OSE’s solutions for communication systems, OSE also offers a safety certified real-time kernel and entire line of safety-critical products, including third party development tools from compilers from DDC-I to modeling tools from Telelogic all optimized for safety critical designs.

OSE for Safety Critical Applications
The basic structure of all of the OSE RTOSs makes fault tolerance very simple to implement - making OSE a natural solution for these types of systems. The safety certified real-time kernel has been implemented in applications such as chemical and process control for industrial automation, or triple redundancy modules (TRM) for safety shut off valves in petrochemical plants.

Highly Reliable Data Management
Also part of OSE’s offering for high availability systems is the Polyhedra In-Memory Database. Polyhedra couples the benefits of an SQL database with built-in fault tolerance, data persistence, and event-driven data change notification. Polyhedra's reliable client-server architecture has been successfully deployed in high performance and mission critical applications worldwide.

OSE’s Direct Message-Passing Advantage
OSE is based on a unique message-passing architecture that provides fast, asynchronous interprocess communication. OSE manages all of the details of buffer ownership as messages are passed from process to process, relieving the application of this responsibility. As a result, applications interoperate much more intuitively. This is the OSE direct message-passing advantage. For more information visit www.ose.com/wp298 to download OSE’s message-passing white paper.

By incorporating OSE’s highly reliable and highly advanced architecture with state-of-the-art software development tools and modern run-time systems, embedded software designers are truly able to benefit from today's advanced embedded processor technologies.

For more information on OSE Systems’ solutions for high reliability and safety critical systems please visit our Web site at http://ose.com/prodserv/safetycrit.

[ Back to Top ]

Tech Talk:

"Unable to Download to Address"
in SCORE®

By Karl Rehmer
Senior Software Engineer; DDC-I, Inc.

Sometimes when a download for running or debugging fails, you will get a message "Unable to download to address xxxxxxxx”. This means that during the load, an attempt to write code or data to the given address failed. A common cause of this is that the address is not a valid address for the actual target. This is often caused by inadvertently linking in an incorrect version of the “User Configurable Code” (UCC). To check what UCC was linked with the application, you can use the “ucc_id” tool. This tool is invoked from the command line as

ucc_id <executable> where <executable> is the name of the program in question.

The response will be something like UCC_Variant_PC for an 80x86 bare PC or UCC_Variant_Excimer for a PowerPC Excimer board.

From the GUI, open a command window (Tools | Command Window) and then give the above command.

The ucc_id tool searches the executable for an identifier that starts with “UCC_Variant” and displays the name. The identifier is defined in the UCC code in the file initialize.c by some code that looks like the following:

/* Dummy variable to enable identification of UCC in executable */ char UCC_Variant_Excimer = 0; 

If you are creating a new UCC for a target board, you should modify the name of the variable to provide an identification for your UCC.

If the response from the ucc_id tool is not the UCC for your target, you should relink the application with the correct UCC.

The method for specifying the UCC from the GUI is found under Project | Tool Options | C Link | Target Options (if the main program is C) or Project | Tool Options | Ada Link | Target Options (if the main program is Ada).

From the command line, the invocation line option depends on the language. For a C main program (the tcc compiler), the option to specify the UCC directory is to specify ‘-Wl:"-U myucc_dir"’ as a target link option or alternatively specify a UCC archive by giving a target link option such as ‘-Wl:"-u myucc.a"’. For an Ada main program, the UCC directory is given to the linker via the option ‘-ucc <directory_name>'. It is also possible to specify the UCC to use by modifying the template file. See the language user guides for details on how to do this.

[ Back to Top ]

Patterns in Postmortems

By Linda Rising

"You must never feel badly about making mistakes," explained Reason, "as long as you take the trouble to learn from them.  For you often learn more by being wrong for the right reasons than you do by being right for the wrong reasons." [Juster61]

Successful organizations today are learning organizations. We know that knowledge is no longer the business of an elite few, but has become the "business of every business." Today knowledge is power, and organizations are struggling with the best ways to manage knowledge, their most precious resource. We also know that just accumulating knowledge is not enough. The real power resides in sharing knowledge, exchanging knowledge among the members of the workplace. Consider the benefits to the software community if something similar to the following were implemented [Davenport+98].

In 1996, teams of leading heart surgeons from five New England medical centers observed one another’s operating room practices and exchanged ideas about their most effective techniques in a collaborative learning experiment. The result? A 24% drop in their overall mortality rate for coronary bypass surgery or 74 fewer deaths than predicted.

I take good care of my health. I exercise and eat right. I try to manage stress. I never plan to have coronary bypass surgery. But if I do—I want one of the surgeons in the experiment! Since these surgeons are all in New England and I live in Arizona, I’m left wondering, "Why didn’t these high-powered doctors write down some "lessons learned" and share those techniques, that decreased mortality rate. That way, doctors all over could improve. It’s the same in software development. We see ways of doing things better but many times we don’t talk about it. We certainly don’t do everything we can to avoid making those mistakes in the future. As a result, we keep making the same mistakes over and over. Here’s the way my friend, Norm Kerth sees it in his book on project retrospectives [Milne26 in Kerth01]:

Here is Edward Bear, coming downstairs now, bump, bump, bump, bump, on the back of his head, behind Christopher Robin. It is, as far as he knows, the only way of coming downstairs, but sometimes he feels that there is another way, if only he could stop bumping for a moment and think of it.

Obviously the importance of knowledgeable people and the sharing of knowledge are not new. Better individual and corporate knowledge has always paid off. What’s new is the focus that is currently being given to this topic. We’re interested in what people know and how to keep them knowledgeable.

One way to address this important topic is to look at what the patterns community is doing. They are trying to capture expertise that resides in the heads of experts, document it, and share it. A pattern is simply a way of documenting a solution to a recurring problem. This method of documentation is an effective way to capture what works so it can be shared with everyone. You can read more about patterns in an earlier article.

Most designers of object-oriented software are familiar with the Design Patterns text [Gamma+95]. Fewer members of the software community are familiar with patterns that have been written for organization and process [Coplien95] but these may be the most powerful patterns of all. Patterns for organization and process talk about what works at the human level. They describe solutions we can all believe in. It’s just that we never wrote them down!

The following appeared in a recent publication [Godfrey99].

In 1988, Joseph M. Juran wrote about the process of deriving lessons learned from retrospective analysis … [and] named this process after philosopher George Santayana, who once observed, "Those who cannot remember the past are condemned to repeat it."

Most large organizations have some form of the Santayana review and may call it a retrospective, a postmortem, a postpartum, or project review. The idea is simple—examine what happened on the last project and learn from it. Capture "what worked well" and "what should be done differently." Project reviews provide a wonderful opportunity to capture knowledge as patterns. In a recent article, I described how to conduct a retrospective. In this article, I’ll describe one way of capturing the knowledge from a retrospective as patterns.

During a retrospective, comments from team members can be captured and documented. The local process owner or quality assurance person should own these comments. In some organizations, knowledge management is supported across projects. Patterns can be written in any of these settings. It does take time and resources. Nothing worthwhile is free. No silver bullet. The patterns don’t write themselves.

Here’s an example of a pattern that was written in one organization. Remember, these are the best practices from other organizations, so pay attention!

Pattern Name: No More Than 10

Problem: How many people should be on a team?

Context: Product development with limited resources and a tight schedule.

Forces

  • Too few people and you won't meet the schedule.
  • Too many people and communication is a bottleneck.
  • You can't always get the people you want.
  • You may have to take the people who are available.

Solution

No more than 10 people should be on a team. If the project is large, you may have several teams. Keep the interfaces between the teams as small as possible to minimize communication overhead. [I’ll say more about large teams in a future article.]

Resulting Context

The small teams can function relatively independently and make the best possible progress toward the delivery date. You’ll always have to fight the urge to add more people. This seems to be the first thing that management wants to do in a crisis. We’re still learning the lesson from Fred Brooks [Brooks95] who observed that adding more women to the job will not produce a baby in less than nine months.

Rationale

Alistair Cockburn states [Cockburn98]:

Separating workers into smaller clusters calls for your architects to partition the system so that teams of 3 to 5 people can [behave] exactly as on a small project.

Fred Brooks states [Brooks95]:

...the ideal...[is] the small sharp team, which by common consensus shouldn't exceed 10 people.

Brooks [Brooks95] also includes the following explanation of Conway's Law (documented as a pattern by Jim Coplien [Coplien95] ):

Organizations that design systems produce systems that are copies of the communication structures of the organizations.

…the organizational structure and the structure of the system are mirror images. Decisions driving the creation of one can affect the other. A clean, well-designed system with minimal interfaces between the sub-systems reflects an organization where each small team works on a sub-system or a well-defined piece of a sub-system, where each team and each piece of the system are relatively independent.

Finally, a comment from Donald Norman [Norman99]:

We work well in small groups, groups of fewer than ten, probably around five.

Known Uses

The following are from project development postmortems:

I try to estimate the amount of work to keep this team of 10 busy. I think the maximum size is 10. More than 10 is too much overhead.

The project had about 10 people on it, average, including System Test and Marketing.

The team had 6-8 developers and 2 system testers, a small team that interacted daily. They sat close together and worked together.

The team was originally 6-7 people. Now there are 30-35. When you have a small group, it's OK to be self-directed. With a larger group, there are too many given the amount of change. It can't be managed. It requires a team of managers.

Team chemistry was very good. There were 5-10 people who worked well together, even those with diverse/clashing personalities.

We kept the team size small, around 9, always less than 10. There might have been pressure from management to add more people to try to get it done faster but we didn't want to add any more people. The 10th person would have made it difficult to divide the work.

Related Patterns

This is a rewrite of Jim Coplien's pattern, Size the Organization [Coplien95]. The new name and the postmortem data make the pattern more compelling.


Here are some abstracts from other patterns that have been "mined" from postmortems along with some of the comments from postmortems that support the patterns. Some of the patterns have been published.

System Test (ST) was involved from the beginning. Testers were just coming off <another project>, so we grabbed them. This worked well for us.

ST and other support organizations should be involved when you first form the team. ST brings a certain reality to the early optimism. Developers are typically narrower in focus and have more specialized expertise. ST brings a broader view, the system perspective.

Get Involved Early [DeLano+98]

Establish a good working relationship with designers early in the project. Don't wait until you need to interact with a designer, by that time it is too late. A good relationship must be built over time. This is true for all support organizations, not just system test.

Our Project Manager is a plus. He is our interface with the outside. We can go to him to resolve issues.

Management’s primary responsibility is to help remove barriers. Bean counting is second.

One person did the scheduling and kept outsiders away. This protected other team members and allowed them to get work done.

Firewalls [Coplien95]

Create a Manager role, who shields other development personnel from interaction with external roles. The responsibility of this role is "to keep the pests away." This pattern fits well with Scrum and other agile processes.

It was difficult to balance the demand for quick turnaround for bug fixes with new feature development. We would get in each other’s way, working on the same code. Sometimes we would have to wait a half-day to get the source. We resolved this by splitting responsibilities. One group would work on fixes and interface with ST. The rest would work on new features.

Team per Task [Cockburn98]

A big diversion hits your team, so let a subteam handle the diversion, the main team keeps going.

It doesn’t have to be a perfect product. If there is a real need and no clear substitute, we can take advantage of customer feedback and improve the product to meet the customer’s needs.

We’re learning that the market won’t pay for higher performance, so maybe poorer (but still acceptable) performance is a better approach.

Good Enough

Work on the product only until it is good enough. Different products require different levels of perfection, depending on customer expectations. This does not imply producing poor quality products. This means producing products that meet or exceed customer expectations but does not involve effort that is not appreciated by the customer.

A critical element in the project’s success was the room with four computers where the team designed and implemented together. This was done from the beginning.

We had a conference room reserved every day 1-3. At the daily Scrum meeting we decided how to use the room. If we needed it, the room was available. If we didn’t need it, we freed the space.

A Place of Our Own

Locate a convenient space near the team and have it available when they need it. A conference room can be reserved or an empty cubicle can be converted to a meeting space. If a conference room is used, try to book the same room at the same time every day. Free it up if it will not be used. The pattern is closely related to War Room and Team Space [Taylor00].

The project had no architecture leader who would make the difficult decisions.

There was no common vision of the software architecture, which caused the system to be designed as two frameworks. The result was over-engineered and too complicated.

Lead Architect [Coplien95]

Define a lead architect role and staff it with the most knowledgeable engineer, who has the most experience and the best leadership skills.

Collocated hardware and software developers helped communication. Proximity makes a difference!

The team occupied two different buildings. The two different groups adopted a different development paradigm. Having software developed in different areas caused communication problems. People were isolated. Project knowledge was aisle-dependent.

I want everybody co-located, too. No fooling around with people in different locations. You never get anything done if people are scattered off every which way. I want them all together. [DeMarco97]

Co-locate the Team

Co-locate team offices.

You might be saying that these seem obvious. Of course they are! At the end of every project, when, because you believe in good software practices, you hold a retrospective, what are those lessons learned? Are they startling revelations? Probably not. They’re probably the same startling revelations you had on the last project, and the project before that, and so on.

What patterns do is document these lessons and give them a name. Then, in the heat of battle, when we’re pushed to fall back on those tried and not-so-true practices like adding more people, someone can say, "Hey! No More Than 10!" and we can remember stories—ghosts from the past—those stories speak to our hearts and can save us. We can recall all the good things we know and maybe this time, we’ll get it right—or at least a little better!

I’m writing two collections of patterns: for conducting retrospectives and for project management (based on experiences with retrospectives). I’d like to hear your stories (everything is confidential, of course). That way, we can all learn. Thanks!

When postmortem data from successful teams shows that a team size of no more than 10 is a factor in the success of the project and when those results are backed up by observations by Cockburn, Brooks, Coplien, and Norman, this is an important pattern. Capturing this important information and giving the pattern the name, No More Than 10, is a useful way to ensure that this knowledge is not lost.

These patterns can be posted internally and as related patterns are collected and organized, they can be presented to the appropriate audience. In this case, most of the patterns in postmortems are for project management (although some are for developers).

Capturing patterns from postmortems is an effective way to ensure that lessons learned about what works and what should be done differently are captured. Since most organizations are concerned about becoming learning organizations, this is a step in that direction.

References

[Brooks95] Brooks, Fred P., The Mythical Man-Month, Addison-Wesley, 1995.

[Cockburn98] Cockburn, Alistair, Surviving Object-Oriented Projects: A Manager's Guide, Addison-Wesley, 1998.

[Coplien95] Coplien, J. O., "A Generative Development-Process Pattern Language," Pattern Languages of Program Design, J.O. Coplien and D.C. Schmidt, eds., Addison-Wesley, 1995, 183-237. http://i44pc48.info.uni-karlsruhe.de/cgi-bin/OrgPatterns?OrganizationBookPatlets

[Davenport+98] Davenport, T.H. and L. Prusak, Working Knowledge: How Organizations Manage What They Know, Harvard Business School Press, 1998.

[DeLano+98] DeLano, D. and L. Rising, "Patterns for System Testing," in Martin, R., D. Riehle, and F. Buschmann: Pattern Languages of Program Design 3, Addison-Wesley, 1998.

[DeMarco97] DeMarco, T., The Deadline, 1997.

[Gamma+95] Gamma, E., R. Helm, R. Johnson, J. Vlissides, Design Patterns, Addison-Wesley, 1995.

[Godfrey99] Godfrey, A., "The Santayana Review," Quality Digest, Feb. 1999, 20.

[Juster61] Juster, N., The Phantom Tollbooth, 1961.

[Kerth01] Kerth, N., Project Retrospectives: A Handbook for Team Reviews, Dorset House, 2001.

[Milne26] Milne, A.A., Winnie-the-Pooh, Puffin Books, 1926.

[Norman99] Norman, D.A., The Invisible Computer, The MIT Press, 1999.

[Taylor00] Taylor, P., "Capable, Productive, and Satisfied: Some Organizational Patterns for Protecting Productive People" in Harrison, Foote, Rohnert:, eds., Pattern Languages of Program Design 4, Addison-Wesley, 2000.


About the Author
http://www.lindarising.org
risingl@acm.org

Linda Rising has a Ph.D. from Arizona State University in the area of object-based design metrics. Her background includes university teaching as well as work in industry in telecommunications, avionics, and strategic weapons systems. She is the author of numerous articles and has published three books: Design Patterns in Communications, The Pattern Almanac 2000, and A Patterns Handbook. She is currently writing a book with Mary Lynn Manns: Introducing Patterns (or any Innovation) into Organizations, to appear in Fall 2002.

[ Back to Top ]



Upcoming
Training:

Introduction to Ada95:
The Language for Safety Critical Applications
Nov. 13, 2002

---------------

FAA DO-178B Basic Seminar
Nov. 14, 2002

---------------

FAA DO-178B Applied Seminar
Nov. 15, 2002

---------------

Seating is Limited!

Click Here for More Info




Check out DDC-I's

Support Program

 

DDC-I Online News is published by DDC-I, 400 N. 5th Street, Phoenix, AZ 85004. Editor Jennifer Sanchez.
 Comments and submission of articles are welcome and should be sent to the editor at the address above or via email at editor@ddci.com.

Copyright © DDC-I A/S and © DDC-I, Inc. August, 2004 - Last Updated August 09, 2004 01:26 PM webmaster@ddci.com