SC11 has just closed and a huge crowd of participants has gone back home with a new input in mind: EMI-European Middleware Initiative.
Our small booth was certainly not a booth one came to visit to get the gadget, but certainly all visitors that stopped by, did have a real reason to linger talking with experts and learn more. This was the aim of this first EMI exhibition at SC11. Let a wider audience know that EMI is the major European Middleware tools and services provider and suggest more users to try our products. More occasions there will probably be to present our next achievements and to meet these new users and the ones to be.
Stay tuned with our news and updates! Meanwhile book a visit to the EMI booth at ISC12 (Hamburg, June 17-21), booth n. 731.
Contributed by Pina Salente.
The third EMI All Hands Meeting, hold in the delightful venue of Padua, Italy, from October 17th to 19th, is over. The meeting has been the occasion for project members to assess the project achievements after its first half, and to consolidate strategies for accomplishment of the objectives left for the remaining period and beyond, with an outlook to the future sustainability of project products after the 18 months remaining.
Full details on the covered topics can be found in the event agenda.
EMI was present at the EGI Technical Forum 2011 to held in Lyon, from September 19 to 23, 2011. The event was hosted by EGI.eu, the Computing Centre of the National Institute of Nuclear Physics and Particle Physics (CC-IN2P3) and France Grilles, the French National Grid Initiative.
The Technical Forum provided an overview of the EGI community and reviewed the community's plans and progress towards the adoption of a federated virtualised infrastructure for European researchers. EMI ran a user tutorial where participants can experience EMI middleware and asked questions to experts. EMI had 2 other sessions: EMI Directions and EMI Security Directions, and contributed to the following face to face :
Data Management Workshop
Virtualization and Cloud Computing: EGI Virtualization Roadmap
EGI Sustainability and Business Models
EMI was also leading a number of sessions in the co-located OGF33, contributing to the co-located SIENA workshop and attending the invitation-only EC eConcertation Meeting. See a detailed report here
Check out the full programme in the Conference web site.
The WLCG Collaboration Workshop in Hamburg on 11-13 July 2011 was an important event for EMI. The workshop was hosted by DESY, one of EMI's partners, and organised by the ever efficient dCache team, one of EMI's software providers. Danilo Dongiovanni and Cristina Aiftimiei, EMI's Testbed Leader and Release Manager respectively, presented the EMI 1 Kebnekaise distribution – why site administrators and users should upgrade to EMI and how to upgrade services which are in production now. That EMI 1 Kebnekaise is the first and only distribution that brings together four of the major middleware providers under one repository with better compliance to open source operating systems and guidelines, that Kebnekaise represents what has been delivered by the middleware consortia separately in a more standard, reliable and manageable way, that Kebnekaise addresses a significant number of outstanding issues in CREAM, DPM, VOMS and WMS, that Kebnekaise delivers a number of new features and more – are some of the reasons why migration to EMI could be attractive to those using and deploying middleware services. Danilo gave a run through on how one migrates from ARC, gLite, dCache and UNICORE to EMI services. Third party validation of EMI products via the EMI Release Preview activity, an opportunity for new and experienced users to preview the EMI release with the technical support of the EMI Testbed team, was also presented.
Zsolt Molnar, on behalf of the EMI FTS product team, presented the new FTS, FTS3.0. FTS3.0 will address a number of issues and limitations with the current FTS implementation – static channel model, oracle backend and complex configuration, among other things. The deployment and development roadmaps of FTS2.x and FTS3.0 were presented. The first FTS3.0 milestone with completely new architecture is expected in April 2012 with no channels, more transfer protocols supported and performance comparable with FTS2.0.
Apart from the welcomed visibility for EMI, the workshop also provided useful feedback and input for the project with respect to middleware services that are in production now and the evolution of WLCG. These include: Overall the middleware services are stable. Middleware documentation is improving. Cristina confirmed the continued yaim support and that no new profiles will be ported to use yaim – comments which were welcomed by some sites. CNRS, via Michel Jouvin, has announced it has quattor profiles for EMI services which will be made publicly available. This will help in the promotion of EMI services to large sites. With regards to batch systems support, Laura Perini of INFN Tier2 Milan offered best-effort help with Condor. Characterising the persistent 10% job failures in production, which has not increased with load, however, addressing it will improve efficiency. SE setup and optimisation are challenging tasks. Can EMI help with the job failure characterisation and lend expert hands to Tier1s' SE setup? There were several references to whole node scheduling to improve resource utilisation and efficiency. Is the middleware ready for this? How much of the middleware is IPv6-ready is one of areas EMI can investigate, following the talk from the HEPIX IPv6 Working Group.
Markus Schulz presented WLCG's deployment and migration plans. WLCG is advising sites and users to migrate to SL5-based EMI services if they benefit significantly. Sites decide, at their own pace, to migrate services with little state data e.g. BDII, CREAM-CE, WMS; those with a lot of state data like Catalogues and SEs should not move unless there is significant gain. Paul Millar clarified dCache components can migrate with no risk of data loss. Move to WN/UI should be delayed for three months pending user and operations verification. There was a clear message from WLCG to EMI to accelerate SL6 deployment and for sites to migrate to SL6 services when these become available. The issue of who drives or dictates deployment planning was raised – is it driven by the NGIs? the sites? user communities? Markus concluded with a number of actions: WLCG will recommend a planned controlled transition for WLCG sites; provide feedback to EMI and EGI; provide information in an official place; "WLCG EMI-1 repository", a controlled copy from EGI will be created.
The mandate of the recently formed WLCG Technical Evolution Group (TEG) was presented. TEG is responsible for defining the evolution of WLCG computing. Its scope, membership, discussion organisation were described. One area TEG will cover is workload management – for instance, how best to formulate and organise requirements for WMS and CREAM-CE, the move towards whole node scheduling and integrated pilot factories. In the ensuing discussion, it has been suggested that membership should not be exclusive and in particular input from middleware providers should be sought. There were encouraging and supportive statements from some of the sites towards EMI and EGI. "Is WLCG taking EMI and EGI seriously?" and "Common services and techniques compatible with EGI (and EMI) is in our interest" were statements from the presentations of Viewpoint from Tier1s and Tier2s.
Co-incidentally, EGI UMD 1.0 was released on the 11th of July. UMD 1.0 is composed of EMI 1 Kebnekaise products. This concludes the transfer and validation of the first release from EMI to EGI. The projects now move to production deployment and support following the recently signed SLA.
Contributed by EMI team at WLCG Collaboration Workshop 2011
The EMI participation at the Joint European DCI Summer School 2011 (DCISS) which took place in Budapest, Hungary from July 11 to 16, 2011, can be archived as a positive experience. The number of student has been 19, coming from different places (Romania, South Korea, Algeria, France, UK, Hungary).
DCI Summer school program was structured in six days, each of which dedicated to one of the projects participating to the school (EMI, IGE, Globus, EDGI, EGI-Inspire, StratusLab).
The EMI day was the first, and has been structured with theoretic presentations in the morning, and tutorials in the afternoon. The two morning presentations have been given respectively by Morris Riedel, covering the current/future project activities and perspectives, while Emidio Giorgio, rather than another presentation, featured demonstrations of some advanced EMI components (Argus, dCache/webDAV).
The afternoon sessions featured an EMI tutorial, actually split in two parts : the first illustrating basic concepts and usage of EMI clients for job submission and data management, while the second part proposed the execution of a complex exercise, requiring the exploitation of all the technologies illustrated in the first part . Coordination of both session has been done by Kathryn Cassidy, supported by Ivan Marton and Zsombor Nagy.
The second part required the execution of a complex exercise, where student had to scan a surface that contained some small bit of relevant information, among a lot of noise. Students had to analyze chunks of the surface it through a Java program, obviously seeking for the relevant bits of information. To facilitate their task, students were given with a hint file, that contained rough indications of the coordinates of the surface where the info were located.
Students had been divided in five groups, and each group was required to find four 'pillars' (the bit of information, represented with a text string). Surface and hints had to be retrieved from a DPM storage element, the surface scanned through job submission towards ARC and UNICORE resources, and once that results had been found, they had to be uploaded to a dCache SE.
The EMI day was closed with a wrap-up discussion between students and Morris, where the key points of the day were discussed and analyzed, being them a valuable input for the next EMI training event.
Pictures (EMI Flickr photostream)
Contributed by Emidio Giorgio
The EMI exhibition experience at ISC11 can be considered a very successful one. We registered a relatively high attendance at the EMI booth where we collected valuable opinions and a positive review from everyone.
We received the visit of the press to which we explained what the EMI project is and what it aims at. So far achievements were as well illustrated. The press was impressed by the difficulty of the project, technically and in terms of human relationships (it's not always easy to ask people/projects to make a step back individually to try a bigger one in a common direction with others!), but amazed by the results so far achieved.
The booth was attended by many visitors who expressed their key concerns on many issues such, as for example, if the merged middleware stack will be able to replace the legacy middleware solutions. Or the sysadmins will be reluctant to do the upgrade. We were also asked if by using EMI one can build LCG resources? Will Tier-1 grid centers be facilitated by the UNICORE components of EMI?
The quest for more details will continue to be satisfied through our web site and the events that the project intends to organize, included some on specific tutorials.
Thanks to all visitors. Visit us again in 2012 as more to be shown will be there!
Contributed by Pina Salente
The second EMI All-hands meeting ended on June 1st in Lund, Sweden. Thank you to all participants for contributing to the success of the event. Meeting agenda and logistics information are still available on the event page.
Pictures from the meeting can be found on the EMI Flickr account.
Bike signs in Lund, Sweden. Photo by Andrea Caltroni
The first EMI Technical Conference will be organised by EMI in conjunction with the EGI User Forum, organized by EGI.eu, Vilnius University and LITNET, with the support of the EGI-InSPIRE project. It will be held in Vilnius, Lithuania from April 11 to 14, 2011.
The event is the first of a series and will be the occasion for all users, developers and technical experts to meet and discuss the achievements of the first year of the project and the milestones for the future. It will be the occasion for an overview of the features of the middleware so far developed and its usability by classic users.
The event will be, furthermore, the place where investigate the possibility for new users to take advantage of the middleware. A "meet the experts and developers" session will give the possibility to users from new environments and peculiar research, industrial and commercial areas to express needs in order to draw a path for a specific customized middleware able to meet particular requirements. An analysis of the middleware sustainability in the future will take place during dedicated sessions.
The event will be co-organized with EGI.eu, Vilnius University and LITNET in Vilnius and will take place along with the EGI User Forum in order to give the chance to users of both projects the opportunity to exchange experiences and know how and develop new collaborations. We look forward to meeting you in Vilnius!
After several weeks of negotiations and refinements, EMI signed a Memorandum of Understanding (MoU) with its largest customer, the European Grid Infrastructure (EGI).
EGI coordinates the network of European National Grid Infrastructures which enable access to computing resources for European researchers from all fields of science.
The MoU was signed on January 27 by EMI's project director Alberto di Meglio and EGI.eu's director Steven Newhouse.
"The software from EMI is critical to the reliable use of our production infrastructure by our user community," says Newhouse. "This MoU identifies the relationship between the two projects for our mutual benefit."
"This MoU will make our business relationship stronger," continues Di Meglio. "It will help us to accomplish our common vision of providing a reliable and sustainable research infrastructure in Europe."
Contributed by Diana Cresti