Simulation is more than Software

Free Trial Version
0 0


High Performance Computing from CADFEM for Satellite Applications

Simulating in new dimensions

The Ferdinand-Braun-Institut, Leibniz-Institut für Höchstfrequenztechnik (FBH), develops reliable semiconductors for applications in the fields of space travel, satellites and quantum technology. The institute covers the entire value chain - from chip design and processing to modules and systems. When it came to procuring new computers for simulations in the field of laser technology, the institute's management turned to the simulation specialist CADFEM.

The efficient interaction of hardware and software is particularly important in the very demanding simulation applications carried out by FBH. One application example is a research facility for quantum optical experiments with ultra-cold atoms, which is to be used on board the International Space Station ISS beginning in 2025. The plan is to analyze fundamental physics issues with quantum objects near absolute zero temperature (-273.15 °C). The core elements of the diode laser modules required for this are laser diodes developed at FBH, which are used together with optics and other passive elements. These laser modules are extremely robust despite their very small dimensions of only 125 x 75 x 23 millimeters.

CADFEM – the competent partner for simulation

Dr. Andreas Wicht, Head of Joint Lab Quantum Photonic Components, explains: “We are a very application-oriented research institute. When designing practice-oriented systems, however, it often happens that information is missing, and we therefore have to examine physical fundamentals in order to better understand and determine the behavior of the systems.” This is why a number of research licenses of the simulation software from Ansys for mechanical, electromagnetic, optical, and photonic analyses are also in use, and why CADFEM was already known at FBH as a competent partner for simulation. “We were not really aware, however, that the appropriate hardware could also be acquired,” reports Dr. Andreas Wicht. “We were happy to be supported by simulation specialists from CADFEM, who very quickly convinced us that a scalable solution would be the right decision for us, since it could be very easily expanded if necessary.”

Christoph Ruß, his colleague from the IT department adds, “we therefore not only invest in hardware from CADFEM, but also purchase services for installation as well as a few hours of IT service for any questions that arise during operation.” Horizontally scalable performance is achieved by combining individual high-performance computing (HPC) servers into groups. This allows very high computing power to be made available to a single user via clustering.

Cluster solution with eleven powerful compute servers

After initial joint discussions between the simulation users from FBH, IT manager Christoph Ruß and hardware specialists from CADFEM, the cluster solution was to consist of eleven powerful compute servers, each with 48 processor cores. Ten were planned with a RAM of 384 gigabytes, and one with a considerably higher storage capacity of 2 terabytes. The latter is mainly to be used for particularly large structures when using Ansys Lumerical.

This software enables photonic multiphysics simulations, meaning the analysis of electrical, thermal, optical and quantum optical effects in a continuous workflow. It was first integrated into the Ansys portfolio in 2020. Lumerical thus complements the broad spectrum of optics simulations offered by CADFEM.

In addition, two virtualization servers for four users and a management server with more than 200 TB storage capacity were planned for the cluster solution. This means that a total of 512 cores are available for computing. For Dr. Andreas Wicht, this is a good concept with which all parties involved are very satisfied, since an efficient interaction of hardware and software could be secured. Christoph Ruß also confirms this assertion and points to the future prospects of this scalable solution, in which additional servers can be integrated very easily. Since another large development project has just been commissioned for FBH, he is already busy planning the expansion.

We need a lot of processing power and memory

To design optimal laser components for space travel and satellite communications, special types of mirrors and lenses, among other things, are used to keep losses as low as possible. Another challenge is the required compactness of the components while maintaining a high level of robustness, since many previous developments have only been suitable for laboratory use, i.e., they are far too large and heavy for use in space. FBH simulation specialist Dr. Igor Nechepurenko reports, “for reliable results, we need to simulate large structures that require as much computing power and memory as possible. We cannot be satisfied with simplified or partial models, because we want to analyze real conditions and behaviors. Many experts were of the opinion that such large structures could not be completely calculated with simulation models. But we have shown that it works with Ansys Lumerical, and we are very satisfied with the results.” With the new hardware, the calculations can be completed in about 24 hours when all the deployable cores (512) are fully utilized. With a single standard server, it would take ten times as long.

The large number of calculations required can be carried out very easily with our new, powerful hardware.

Simulation specialist Dr. Igor Nechepurenko

How can the required compactness and robustness be achieved?

In order to develop ever smaller components and make the entire structure more compact, many traditional thermomechanical simulations are also carried out with Ansys. The heating in this case is not the actual problem, but rather the resulting deformations that are very important. “For optical assemblies with a longitudinal scale of one micrometer, which corresponds to the wavelength, a thermomechanical deformation in lens or mirror components of only one micrometer would already mean the end for the optical unit,” emphasizes Dr. Igor Nechepurenko. “Consequently, we have to simulate very precisely in order to analyze the thermal limit to which we can go.”

Dr. Andreas Wicht adds, "at FBH, we are working in parallel on an even more compact variant and are currently transferring the proven concept of the hybrid-built Extended Cavity Diode Laser, or ECDL for short, to a single chip. With this new concept, which is implemented entirely by semiconductor technology, future laser units will not only be smaller and more robust, but also much more affordable." However, this means realizing more functionality on one optoelectronic component (laser chip).

Previous methods are no longer sufficient

Previous design methods alone, however, are no longer sufficient for this. This is because they are often based on the extensive experience of colleagues in laser dimensioning that has been gathered over years and decades. This makes it possible to keep an eye out for a few parameters and improve certain designs. But this does not work if a large number of parameters determine the design and, for example, topology optimizations are to be carried out. Here, virtual models are needed that not only represent the principle mode of operation, but also enable developers to precisely predict the real behavior of variants that might differ. “We are currently checking how precise our calculations are,” explains Dr. Igor Nechepurenko. “To do this, we repeatedly perform new calculations with slightly modified settings in the software so that we can subsequently analyze and compare the results obtained with them in detail. This multitude of calculations is very easy to perform with our new, powerful hardware, which was installed a few months ago.”

Remote installation and IT support

IT manager Christoph Ruß points out that this installation had to take place under difficult conditions. This is because Klaus Fischer, one of CADFEM's hardware specialists, had to coordinate and carry out the installation of the entire server cluster during the pandemic, without being on site himself. The communication components integrated into the installed HPE servers enable the computers to be operated completely remotely. This made remote support as well as remote installation easier, although Christoph Ruß was able to complete the necessary “manual work” from FBH's on-site IT service.

“Not only does Klaus Fischer have a great deal of know-how and extensive experience, but he also has the ability to explain things extremely well,” says Christoph Ruß. “This made our cooperation very easy and pleasant. For example, he was able to pack any desired computer configuration image into the virtual drive in order to install it directly on the respective server and then check the required functionality.”

Not only does Klaus Fischer have a great deal of know-how and extensive experience, but he also has the ability to explain things extremely well,” says Christoph Ruß. “This made our cooperation very easy and pleasant.

Christoph Ruß, FBH IT Service

You have to speak the same language

Klaus Fischer’s presence on site was not absolutely necessary, but of course, he had to have an idea of how the on-site installation had to be carried out. It was also necessary to consider the skill level of those who were on site. “This was no problem for Klaus Fischer,” explains Christoph Ruß. “He took a very focused and professional approach. I found the calm manner in which he explained the work that needed to be done and conveyed the relevant background knowledge to be very pleasant. It quickly became clear that we spoke the same language. This created trust, which gives us security and is an important basis for a long-term successful cooperation.”

Further expansion is already underway!

The next step in this cooperation is currently being taken, namely the further expansion of the cluster solution by another eleven servers, bringing the number of cores to 1040. Considering that simulation specialists at FBH only started about 20 months ago with a significantly lower number of cores for Ansys Mechanical and Electromagnetics applications, this is an enormous leap forward. The higher number of cores not only benefits the Lumerical application, but of course also the traditional Ansys applications. This leads to shorter computing times and makes it possible to analyze more or larger models.

In conclusion, Dr. Andreas Wicht emphasizes, “we have the feeling that we are always in very good hands with CADFEM. This applies to both to the software we need for our simulations as well as the hardware, which we have acquired and successfully installed in recent months with CADFEM's support. From the user side, Dr. Igor Nechepurenko, among others, is very satisfied with CADFEM's support. From FBH's internal IT side, Christoph Ruß equally compliments CADFEM for such a pleasant cooperation.”

We have the feeling that we are always in very good hands with CADFEM. This applies to both the software and the hardware.

Dr. Andreas Wicht, Head of Joint Lab Quantum Photonic Components

Leibniz-Institut für Höchstfrequenztechnik (FBH)
Dr. rer. nat. Andreas Wicht,
Head of Joint Lab Quantum Photonic Components

Author: Gerhard Friederici (CADFEM Germany)
Images: © Ferdinand-Braun-Institut, Leibniz-Institut für Höchstfrequenztechnik (FBH)
Published: March, 2023

Contact CADFEM

Area Manager IT Services