LondonGrid

RHUL cluster expands

2011-03-11T12:41:00.003Z

Yesterday, RHUL took delivery of new storage and compute nodes to beef up its Tier2 cluster.
The GridPP and CIF funded kit was supplied by Dell and is being installed and configured by Alces.
The extra 6.3 kHS06 and 420 TB will more than double the capacity of cluster.
Once the installation is complete and accepted, work to integrate it with the existing cluster and bring up the gLite services will begin.

RHUL 'Newton' cluster comes home

2010-02-19T17:29:00.002Z

After two years hosted by Imperial College, our 'Newton' Grid computing cluster has finally been relocated to Royal Holloway's new state-of-the-art computer centre. The move was carried out by Clustervision and everything went smoothly. Before the cluster goes back into production, analysing LHC data, a software upgrade to SL5 is planned.

A small part of Newton remains at IC: the racks were donated to become part of the particle physics cluster.

Comparing ATLAS analysis at RHUL using the file-staging and RFIO approaches

2009-07-31T12:14:00.028+01:00

I have been looking at the performance of the Royal Holloway cluster during Hammercloud tests in which data was accessed directly from the DPM pool nodes using the RFIO protocol and comparing it to the recent UK-wide file-staging test (540).

For the RFIO approach two identical tests (537 and 538) were requested in order to ensure enough jobs arrived on site. The RFIO IOBUFSIZE was set to 4KB. Job CPU efficiencies and cluster throughput (the product of number of running jobs and average job efficiency) were extracted using Sam and Dug's script. The job throughput climbed steadily up to a peak at around 320 running jobs. At this point the throughput started to decline probably compounded by the fact that one of the disk servers lost a disk and became over-loaded.

The CPU efficiency declined relatively consistently as the number of running jobs increased:

Each job was reading data at about 1 MB/s so that at the peak the total bandwidth was around 350 MB/s - roughly 30 MB/s per disk server. The disk servers were working hard, however, the iostat %util values were around 100% with high cpu iowait values.

So how do these results compare to those obtained when staging files to the worker node prior to analysis? This graph shows the same RFIO throughput data together with results from the recently run file-staging test:

The throughput during file-staging leveled off earlier - at around 175 running jobs. Similarly the average job efficiency drops more steeply:

The job failure rate for the RFIO tests was 4% compared to 17% for the file-staging test.

RHUL getting good rates into MCDISK from RAL

2009-05-13T19:55:00.004+01:00

RHUL has regularly got good rates and by that I mean 80-100 MB/s from Fermilab when downloading CMS data. It nice now to see similarly high rates downloading ATLAS data into the MCDISK space token from RAL.

Exercised space token creation at UCL-HEP

2008-04-16T13:44:00.003+01:00

Thought it was neat to give it a try and created as a test a small reservation for dteam, following the instructions on the LCG Twiki. All went well and all the tweaks for SL3 / gLite 3.0 worked well. Only oddity was that:

[root@pc55 root]# dpm-reservespace --gspace 10M --lifetime Inf --group lcgdteam --token_desc dteam_10M
send2nsd: NS009 - fatal configuration error: Host unknown: UNUSED
invalid group: lcgdteam

but:

[root@pc55 root]# dpm-reservespace --gspace 10M --lifetime Inf --gid 2688 --token_desc dteam_10M

worked well. Perhaps due to the fact that the group id is not the same as the VO name?? (tried also with 'dteam' in place of 'lcgdteam', but had the same error.

RHUL aircon problems

2008-03-26T14:41:00.003Z

Our machine room aircon system broke down last week and the temperatures have been all over the place.

After a few days of summer clothing and a few nights of temperature alarms, it was diagnosed to be a refrigerant gas leak from the chiller on the roof. The bad news is that this takes 2 weeks to fix. Luckily the estates engineer was very efficient and organised the delivery and connection of a backup chiller on the last day of term, then personally looked in over Easter to keep an eye on it.

It has been stable the last few days so I've just brought the cluster back up. The site will come out of downtime this evening.

UCL-HEP APEL accounting fixed

2007-08-07T16:15:00.000+01:00

After upgrading to gLite r27 on the 4th of July, APEL stopped publishing to the central RGMA registry. The apel-publisher failed with a not handled

RGMABufferFullException

To fix this, we had to update to the latest version of the APEL rpm's (2.0.5-1) on the MON and CE and re-run YAIM on both

Imperial SE - dCache removed ~30TB of CMS data

2007-07-20T14:00:00.000+01:00

As requested by CMS users, this week we have cleaned up around ~30TB (orphaned files) of CMS data from IC dCache. We need to understand why so many orphaned files are generated in dCache.

Brunel SE running DPM 1.6.5

2007-07-20T13:19:00.001+01:00

We were having problems with the storage element at Brunel so I upgraded it to DPM version 1.6.5 (via 1.6.3) this week. The upgrade didn't go totally smoothly but now things seem a lot better. Thanks to Greig for his usual excellent support.

Brunel running SL4 cluster

2007-07-20T12:46:00.000+01:00

The worker nodes of dgc-grid-40 are now running the glite worker node release on SL4. It is passing the ops SAM tests and the VO tests that have run recently. There was a problem with LHCb production jobs trying to use edg-brokerinfo rather than glite-brokerinfo which I reported and they have now fixed. CMS production jobs have also completed successfully. Steve Lloyd's ATLAS tests pass apart from the 'New Package' part. Steve's comment was "My tests are still running release 12.0.6 for which the requirement is SL3 so they shouldn't really go into SL4 machines...this problem will go away when I switch to release 13.0.X as that's supposed to work on SL4". ATLAS production jobs seem to run OK but there seems to be a problem copying the output files back.

RHUL accounting problem

2007-07-20T12:32:00.000+01:00

There was a problem with the apel accounting at RHUL this week:

ZoneInfo: /usr/java/j2sdk1.4.2_12/jre/lib/zi/ZoneInfoMappings (Too
many open files)
Thu Jul 19 00:35:06 GMT 2007: apel-pbs-log-parser - WARNING -
Exception opening file /var/spool/PBS/server_priv/accounting/20070713
java.io.FileNotFoundException:
/var/spool/PBS/server_priv/accounting/20070713 (Too many open files)

we solved it by moving some of the files out of /var/spool/PBS/server_priv/accounting.

bdii counts

2007-06-26T05:53:00.000+01:00

Promised to monitor the bdii. This is the plot of the bdii count a while ago. I'll have to redo it for a longer period. It seems clear that it is not the entire site bdii that disappear but only individual entries. Which is very probably correlated with load. We have seen it with the ce mds.

RB very slow

2007-06-19T10:28:00.000+01:00

Yesterday I have been wrestling with our RB. I takes several hours for a job to go from waiting to scheduled which means that the matchmaking process is overloaded. I think the reason was that the database was very big (4GB). Exacly 2^32. As suggested here I cleaned the database and it seems better now. The problem is that I never got to the root of what was going wrongly...

dCache failures (dcache-server-1.7.0-36)

2007-06-19T10:22:00.000+01:00

Again this morning we have pools going down with a memory allocation problem:
--
06/19 00:45:58 Cell(sedsk01_5@sedsk01Domain) : Thread : ping got : java.lang.OutOfMemoryError: Java heap space
--
I think the only way we will solve this will be to get hold on a dCache developer that can have a look. Clearly we did not have this problem when we where running the previous version (release 35).

dCache pools went down

2007-06-18T13:25:00.000+01:00

From friday afternoon several dCache pools went down. It ran out of memory, and here is the content of the sedsk01Domain.log file.

06/15 16:32:13 Cell(sedsk01_1@sedsk01Domain) : at java.lang.Thread.run(Thread.java:595)
06/15 16:32:13 Cell(sedsk01_1@sedsk01Domain) : Storing incomplete file : 0003000000000000006E0B80 with 2756018417
06/15 16:32:13 Cell(sedsk01_1@sedsk01Domain) : Stacked Exception (Original) for : 0003000000000000006E0B80 <-P---------(0)[0]> 2756018417 si={cms:cms} : CacheException(rc=10006;msg=Pnfs request timed out)
06/15 16:32:13 Cell(sedsk01_1@sedsk01Domain) : Stacked Throwable (Resulting) for : 0003000000000000006E0B80 <-P---------(0)[0]> 2756018417 si={cms:cms} : CacheException(rc=33;msg=Illegal State Transition -P-------- -> -P--------)
06/15 16:32:13 Cell(sedsk01_1@sedsk01Domain) : CacheException(rc=33;msg=Illegal State Transition -P-------- -> -P--------)
06/15 16:32:13 Cell(sedsk01_1@sedsk01Domain) : at diskCacheV111.repository.CacheRepository2$CacheEntry.setPrimaryState(CacheRepository2.java:107)
06/15 16:32:13 Cell(sedsk01_1@sedsk01Domain) : at diskCacheV111.repository.CacheRepository2$CacheEntry.setPrecious(CacheRepository2.java:219)
06/15 16:32:13 Cell(sedsk01_1@sedsk01Domain) : at diskCacheV111.repository.CacheRepository2$CacheEntry.setPrecious(CacheRepository2.java:215)
06/15 16:32:13 Cell(sedsk01_1@sedsk01Domain) : at diskCacheV111.pools.MultiProtocolPool2$RepositoryIoHandler.run(MultiProtocolPool2.java:1538)
06/15 16:32:13 Cell(sedsk01_1@sedsk01Domain) : at diskCacheV111.util.SimpleJobScheduler$SJob.run(SimpleJobScheduler.java:64)
06/15 16:32:13 Cell(sedsk01_1@sedsk01Domain) : at java.lang.Thread.run(Thread.java:595)
06/15 16:35:02 Cell(c-100@sedsk01Domain) : runIO : java.lang.OutOfMemoryError: Java heap space
06/15 16:35:02 Cell(c-100@sedsk01Domain) : java.lang.OutOfMemoryError: Java heap space
06/15 16:35:02 Cell(c-100@sedsk01Domain) : java.lang.OutOfMemoryError: Java heap space
06/15 16:38:25 Cell(c-100@sedsk01Domain) : runIO : java.lang.OutOfMemoryError: Java heap space

dCache is started with those parameters:
-server -Xmx512m -XX:MaxDirectMemorySize=512m

We don't know what happened.

Dataset access problem at IC-HEP

2007-06-15T03:47:00.000+01:00

Some users are experimenting datasets access problems at IC-HEP. The ticket in question is GGUS 22106. The problem is that our cms users don't have the problem for the same dataset.
This raises the question on how to debug those problems when you don't have users on hand. In this case the only solutions will be to do it interactively with the user.

SAM Failures in London

2007-06-15T03:28:00.000+01:00

Summary of SAM failures and solutions

mars-ce2: CA certificates updated but permissions where wrong for the lt2-lcg group and hence the certs where not readable. Fixed now
hep-ce:

Update of the images. Missing ssl and uuid libraries caused the lcg-cp tools to fail. Matt solved this
updated the CA but unfortunatly the crl cronjob did not run since it is being run by mona. Now fixed

gw-2 (UCL-CENTRAL): Investigated intermittent failures and discovered that the sam jobs are sometimes killed by sge which has a vmem limit of 2GB. The problem is that python when creating a new thread tries to use the max stack size of the parent process. Since sge set this with a very high value any new thread will thread will try to create a big stack and the vmem limit will be reached. The solution is to change the max stack size in the sge configuration. We tried a ulimit -s 10 in the jobmanager but since then gw-2 is failing the ops test consistently. William has been contacted the revert back this change and make the modification in the sge queue configuration.

Note: this problem was seen on the ic-hep cluster (ce00) and fixed using the stack size limit.

ce1.pp (RHUL): gatekeeper problem, it seems I cannot access with the ssh keys I am using at home. Have to check from IC.

It's a black week for the availability in London...

London Tier2 Workshop

2007-05-02T12:03:00.000+01:00

The London Tier2 Workshop took place on the 16 of April.
It was a good opportunity to see what are the non hep application running on the grid.
The slides of the workshop can be found here

New Grid Security Policy Document

2007-05-02T10:37:00.000+01:00

The new Grid Security Policy Document can be found at here . It is still a draft, and comments are welcome. See version 5.6

RB Wrestling the comeback

2007-02-20T10:55:00.000Z

This morning looking at the monitoring our RB does not look happy. You can judge yourself on the plot below. It clearly seems that when the submission rate is too high the workload manager can just not eat the jobs fast enough to reduce the queue length. I have asked help from Maarten, we'll see what he come up with. I think I will have a look in the rb code to find out what is going on...

Certificates and Mars

2007-02-19T17:08:00.001Z

I was worried by the low number of jobs at LeSC. There was not much jobs there.
It is very difficult to get an hold on the output files of failing jobs. Thanks to sge we can find out where it is located

qstat -j jobid will print out the std.err and std.out of the jobid given

The problem was:

globus_i_gsi_gss_utils.c:2155: globus_i_gsi_gssapi_init_ssl_context: Error with openssl: Couldn't open bio for reading on file: /homes/lt2-lcg/grid-security/certificates/47d3d1a0.0

and that is because when untarring the files in the certificate directory one of the certificate
was not readable by the lt2-[users]. This is now fixed and I will chase up lhcb to understand
if they can run there without problem.

Wrestling with our RB

2007-02-19T16:56:00.000Z

We are still observing very long time (several minutes) to have a job going from the waiting state to the scheduled state. This means that the network server of the rb is accepting the job but the workload manager is running out of steam to process it and do the match making.

I monitored the rb by looking at the number of entries in the input queue (/var/log/edgwl/workload_manager/input.fl). Checked the number of entries that matches the regular expression ("g$").
Plotting the number of entries waiting to be accepted by the workload manager as a function of time. The result is here.

The left scale (blue dots) is the number of jobs waiting to be matched. The right scale (red dots) is the number of jobs submitted per unit of 10 minutes

You can see a clear drop at the end of the x range. I think this is because I have reduced the number of threads for the network server and increased that number for the workload manager. The file to look at is /opt/edg/etc/edg_wl.conf .

For the NetWorkServer:

MasterThreads = 4;
DispatcherThreads = 6;

For the WorkLoadManager

NumberOfWorkerThreads = 10;

I will continue to monitor it during the night because the drop is not fully understood. Maybe it is the cms production that has slowed down and is giving some air to the rb.

Resurrection of the Blog

2007-02-12T13:14:00.000Z

ICT the Grid

Today we are back in business with ICT to get their cluster on the Grid.

They will provide one machine and install RHEL3 i386 so that we don't have the RHEL4 problem.
We have to find out how to modify the information system since they are running pbspro which does not have exactly the same commands as pbs.
They will create the pool accounts and we have yet to make sure that we can run prolog scripts to get the lcg environment correct

QMUL

Atlas cannot install the tags. They tried to install the new software but it is not published correctly.
Maybe this is because we are publishing another subcluster to publish the 64 bit queues. I'll make a wiki entry with explanations how this was done. The dynamic information does not seem to be correct though.

Imperial Hep

Mona has enabled camont and total on our rb. We now need a site to test it and we also need to enable it on the lesc and ic-hep ce.

dzero

2006-12-04T10:24:00.000Z

- Dzero was disabled at QMUL because the NAT box changed. This is now
solved and we see dzero jobs at QMUL again. Next will be
ce00.hep.ph.ic.ac.uk
- CMS installation should now be possible at QMUL. We are trying to get
all the London sites cms installable by the sgm.

mars will be offline

2006-12-01T11:32:00.001Z

UKI-LT2-IC-LESC will be down from the 1/12 - 4/12 for maintenance.

SAM, NGS

2006-11-09T09:40:00.000Z

Yesterday we have found why we had Aborted SAM tests. The reason was simple, the SAM jobs seems to use a lot of virtual memory and they where killed by our lovely SGE. We where unlucky since all the tests could be published and where showing ok. Only the RB state was showing an Abort with a Maradona error. The key to find why the jobs where failing was to use the sam tests ourself with the information found here.
Concerning NGS the setup is done at LeSC. But we have a problem with the home directories that are not mounting properly on the worker nodes. Keith is looking at it.
Another usefull info is that we do not have to setup a specific UI for the NGS. They can submit from their UI and they use the stage in switch of globus-job-submit (globus-job-submit mars-ce2.mars.lesc.doc.ic.ac.uk:2119/jobmanager-sge -q 10min -s `pwd`/test.sh)
APEL accounting, Duncan and Giuseppe are chasing problems with the data not being published.
LHCB installation at Imperial failed, Joel is trying another time
Today we reached more than 1.8k jobs in London, Thanks everyone and welcome to the new clusters. Brunel is running full steam and Imperial is catching up.

2006-09-25T19:29:00.000+01:00

UCL-CENTRAL: Today I chased up the replication sft problem. I don't have any answer from my ggus ticket (#12970). William has mapped me to opssgm and I can sucessfully store files on their srm. When looking more in detail it seems that they are also openssl errors spitted by the server. They only appear when it is the "regular sft" that are running. If we use the polish portal we don't have the error. My bet it that the portal uses grid-proxy-init while the "regular sft" is using voms-proxy-init. It is probably because the vomses and vomsdir content is not completely correct. William is having a look at that.

ATLAS: I have contacted Alessandro to understand why some of the London sites are not in shown on their production portal. He explained me that they have to populate a list of sites to which they submit production jobs. I will make sure with Kondo that the atlas sw is there on all london sites.

CMS: The cms production is finished (50 millions events). CSA06 should start in the coming weeks to analyse the produced data that is fed back to CERN.

Woodcrest: Today we (me, Mona, Bill) physically installed all the machines we currently have (22 wn and 8 disk servers).

PPS: Barry is summarizing all the problems he went trough with the glite WMS. Hope to have this for the next operation meeting.

dCache: Having problems with the number of connection in close_wait state. This is a known issue to the dCache team. We don't know when it will be fixed. This causes replication failures ...

Transfer test: RALPPD-->QMUL transfer test worked. Average bandwidth 106Mbits/s and 1146/1500 files transferred. Clearly we should be able to do better since they have a 1Gb link which is not much loaded.

GridMon: Got an agreement with Mark to have the recepie to rebuild the GridMon box for Imperial and UCL. Kostas agreed to build it.

Brunel new cluster:Discusssions with Duncan on how to proceed with the new cluster.

2006-09-22T15:48:00.000+01:00

Today is a horrible rainy day... I have been again busy with QMUL. For some weird reason lhcb has managed to schedule 2500 jobs there and none are running. I can even not see the jobs in the queue. I restarted the gatekeeper and I can see jobs being submitted by Ricardo. I still don't understand what was happening there. I have also tried to ressurect the ganglia of QMUL which is down. I could not power cycle it.

Transfer Tests: I have initiated a transfer of 100Gb from RALT2 to QMUL it seems to be fine for now.

Woodcrest: 20 new machines arrived today.

2006-09-21T16:02:00.000+01:00

Woodcrest: Asked ICT to reserve a set of UID/GID to avoid clashes between hep and ict.

Biomed Challenge: Asked Yannick to see if we can find another solution then inbound connectivity to the nodes for the flex licence.

QMUL: try to understand what causes such a low number of running jobs while there is a lot of jobs queued. see attached plots.

DNS problem: very high load on the dns server. Process zombie when trying to kill it. All Grid services stuck. Could restart the dns and the situation seems to be stabilized.
Maui conf: Reservation did not work if ops jobs where submitted on the long queue. This is because the reservation period was not set to infinity.

Dzero:

dzero station sandbox full. All dzero sites affected. Frederic was waiting that QMUL is back to send jobs there for testing.
I have investigated lesc to try to understand what causes such a low number of running jobs while there is a lot of jobs queued. see attached plots. On the left number of scheduled jobs and on the right running.

Other Stuff:

SAM Monitoring: Asked all sites to check their status in SAM
Asked to be mapped to ops to test UCL srm. No answer yet to the ticket #12970
Installed new apel accounting rpms at UCL-HEP. Will see tomorrow if it has solved the problem.

2006-09-20T09:38:00.000+01:00

QMUL:

Certificates have not been upgraded, the tests are now critical and Giuseppe is on Holiday up to the 27/09. I will upgrade them since Giuseppe is on holiday. Have made a new rpm version 8 for the certs. Have a look at our wiki
lhcb yaim settings have been changed to match their use of groups rather than roles. It seems ok and I am populating the LT2 yaim file with the corrected entries

Biomed challenge: Have asked all London to update the biomed voms to be ready for the biomed challenge of 1/10. Will have to fill in the ressource pledge in the cic portal here. Duncan has checked with Yannick that the voms settings is ok.

NetMon: Trying to find a solution for the net monitoring box proposed by Robin. Admins wants to rebuild the machine. I am afraid this will take time to be resolved.

Woodcrest purchase: Today we have received 8 new boxes. The disks one. We have also debugged a problem with the pxe boot. It seems that some of our switches are not behaving correctly since the dhcp requests do not come back. Kostas is investigating further. For the CE we will build an old machine. We are ordering additional 1Gb memory for the CE that will go to ICT and HEP.

Dzero: Frederic has tested dzero at QMUL, it seems that the setup we have made for the TMP directory is ok. He is having concerns with the number of slots he is having at LESC.

2006-09-19T09:09:00.000+01:00

QMUL: poolfs is down again. It says no space left on device. Restarting poolfs does not help. I have decided to reboot the storage element. The reason there was a poolfs problem is that the / was full. I have setup the logrotate for the logs to be compressed which is not the case by default in the logrotate file created by yaim. The se01 is back online.

All Hands 2006: Prepared an small talk "Towards Sustainability" to initiate discussions.

UCL-CENTRAL: still having problems with replication for the ops vo. Wiliam observed that if he runs the test using the submission tool it does not give a replication error. Submitted a ticket (#12970) to understand the differences.

IC-HEP: Viglen as delivered the racks and two machines. We are preparing to install them.

RHUL: Going down for the week-end.

2006-09-18T09:40:00.000+01:00

SC4 CMS: Last friday Brunel was validated for CMS production. It is now running full with cms jobs. We had to play with the dpm permissions to make this work and now doing the same at RHUL. I am trying to keep the London SC4 activity page updated. See below a plot taken on the 18/09/2006 showing the Brunel activity.QMUL:

poolfs died, the index is not reachable. It seems that autofs is not behaving correctly. I did not dare to restart the machine remotely and I made a hard mount to the index under /tmnt/poolfs. dpm is back.
Prepared qmul for cms sc4 . Waiting for the cmssw to be installed and will have to make the dpm hack.

LeSC: Frederic is having problems with dzero jobs. A lot of jobs are scheduled but they seem to dissapear. See the two plots below. I suspect that some of the jobs have a very short time because of a stager error and they are not seen as running since they last for less than five minutes. We will check in the root file to see what wall clock time distribution the job have.

IC-HEP: Our rb is creating a lot of proxy renewal and filling up the disk., Mona is investigating.
We have received two woodcrest from Viglen and I need to check with the order.

UCL-CENTRAL: Alice left thanks for her work !
We have problems with the ops vo, the replication does not work. William saw that it is due to the default se that is not set correctly. Probably a bug in yaim. It should be fixed now.

Activity restarted after Holiday

2006-09-11T09:49:00.000+01:00

I have tried to recover from my emails.

I have started to gather the slides/data for the rtm talk of friday Service Challenge Technical Meeting .
dCache head node was down due to a full /var. Mona cleaned the directory and started to look at the nagios dCache scripts
Duncan is off ill and I will have a look at the RHUL sft failure
Started discussing with Matt Harvey (ICT) about the pbspro they are using. We will have to adapt the job manager to that version
Gave contact details to Mark Leese for the Net box to have one at Imperial and asked William if he is happy with having such a box at UCL-CENTRAL
Giuseppe wanted help to create 100*1Gb files in his dpm. suggested him to use rfcp from one of his cn nodes.
Jamie told to hold since Alice was performing UCL-CENTRAL tests.
Followed the castor/d-cache phone conf that was held at RAL.