Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .github/workflows/tutorial-ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
name: tutorial-ci

on: [pull_request, workflow_dispatch]

jobs:
check:
uses: eic/tutorial-ci/.github/workflows/check.yml@main
42 changes: 21 additions & 21 deletions episodes/02-rucio_usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,14 +59,14 @@ The name contains information about the dataset in question and contains informa

As an example, consider the DID for the dataset:

- `epic:/RECO/26.02.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x130/q2_10_20/pi+`
- `epic:/RECO/26.03.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x130/q2_10_20/pi+`

The `name` here - `/RECO/26.02.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x130/q2_10_20/pi+`, tells us many things about the contents of this dataset. Let's break this down, examining the component enclosed within each pair of `/---/` -
The `name` here - `/RECO/26.03.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x130/q2_10_20/pi+`, tells us many things about the contents of this dataset. Let's break this down, examining the component enclosed within each pair of `/---/` -

- `RECO`
- This tells us that the DID contains reconstructed output file information
- `26.02.0`
- This tells us that the `26.02.0` software release was used, the February 2026 release (version 0).
- `26.03.0`
- This tells us that the `26.03.0` software release was used, the March 2026 release (version 0).
- `epic_craterlake`
- This tells us that the `epic_craterlake` detector configuration was used in the simulation
- `EXCLUSIVE`
Expand Down Expand Up @@ -129,15 +129,15 @@ Remember that campaigns older than ~6 months will not be instantly accessible. T
Working backwards from the full DID we had earlier, we could combine in the software release, detector configuration, process and generator to narrow down the list of DIDs:

```bash
rucio did list epic:/RECO/26.02.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/\*
rucio did list epic:/RECO/26.03.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/\*
```

We now have a more manageable list and we can see that we have some different beam energy configurations and Q2 ranges available. Note that we can list just the DID itself by adding `--short` as a prefix to our `did list` call.

We could also make slightly better use of wildcards in this command too:

```bash
rucio did list epic:*RECO*26.02.0*DEMP*
rucio did list epic:*RECO*26.03.0*DEMP*
```

But as above, this does require some knowledge of what our DID looks like to begin with.
Expand All @@ -161,19 +161,19 @@ rucio did content list scope:name
e.g.

```bash
rucio did content list epic:/RECO/26.02.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+
rucio did content list epic:/RECO/26.03.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+
```

Again, a lot of output. We see though that this DID is for a dataset of a large number of files. We can again use `--short` to just get this as a list:

```bash
rucio did content list --short epic:/RECO/26.02.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+
rucio did content list --short epic:/RECO/26.03.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+
```

We can check where a specific file in our dataset is stored too:

```bash
rucio replica list file --protocols root --pfns --rses isopenaccess epic:/RECO/26.02.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+/DEMPgen-1.2.4_10x250_pi+_q2_3_10_ab.0550.eicrecon.edm4eic.root
rucio replica list file --protocols root --pfns --rses isopenaccess epic:/RECO/26.03.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+/DEMPgen-1.2.4_10x250_pi+_q2_3_10_ab.0000.eicrecon.edm4eic.root
```

::::::::::::::::::::::::::::::::::::::::::::: callout
Expand All @@ -183,21 +183,21 @@ rucio replica list file --protocols root --pfns --rses isopenaccess epic:/RECO/2
Despite the slightly misleading command above, we can actually just provide a dataset DID here too. If we do so, we will get the location of all files in the dataset in one command, e.g:

```bash
rucio replica list file --protocols root --pfns --rses isopenaccess epic:/RECO/26.02.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+
rucio replica list file --protocols root --pfns --rses isopenaccess epic:/RECO/26.03.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+
```
You could pipe this to a file for later usage. However, note that replicas may exist for a given file. Both would be printed by this command as is. You can check if multiple copies exist via:
```bash
rucio rule list --did scope:name
```
e.g.
```bash
rucio rule list --did epic:/RECO/26.02.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+
rucio rule list --did epic:/RECO/26.03.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+
```
This will list where the DID is stored and how many copies exist.

:::::::::::::::::::::::::::::::::::::::::::::

The `root://dtn-eic.jlab.org` at the start of the output tells us that this particular file is stored on JLab servers. As mentioned in the outset, Rucio works across multiple sites easily, however, methods which we might use to stream files do not. **As such, being able to check where our files are stored is a useful feature.**
The `root://epicxrd1.sdcc.bnl.gov` at the start of the output tells us that this particular file is stored on BNL servers (files stored at JLab start with `root://dtn2304.jlab.org`). As mentioned in the outset, Rucio works across multiple sites easily, however, methods which we might use to stream files do not. **As such, being able to check where our files are stored is a useful feature.**

So, we can find DIDs, check what they are and what they contain. To get to this point though, we needed some pre-knowledge of what the DID looked like which isn't necessarily that helpful for finding something. However, a much easier approach to finding what we need is to use the metadata tags that are assigned all DIDs from March 2026 onwards.

Expand Down Expand Up @@ -343,7 +343,7 @@ rucio did list --short --filter 'software_release=26.*, generator==dempgen, elec
```

This should return the matching DEMP dataset DID (of the form
`epic:/RECO/26.02.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+`). Feed that DID
`epic:/RECO/26.03.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+`). Feed that DID
into `rucio did content list --short scope:name` and count the returned lines to get the number of
files in the dataset.

Expand All @@ -364,7 +364,7 @@ rucio download scope:name
So to download our file from earlier, we just do:

```bash
rucio download epic:/RECO/26.02.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+/DEMPgen-1.2.4_10x250_pi+_q2_3_10_ab.0550.eicrecon.edm4eic.root
rucio download epic:/RECO/26.03.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+/DEMPgen-1.2.4_10x250_pi+_q2_3_10_ab.0000.eicrecon.edm4eic.root
```

By default it will download to our current directory with its original name. In this case, that's unfortunate because as we noticed earlier, this looks a lot like a UNIX file path. As such, we now have a large number of nested directories to go through before we get to our file!
Expand All @@ -384,23 +384,23 @@ We can *stream* files from a full dataset rather than downloading them as we'll
It might actually be easier to use [XrootD](../learners/xrootd.md) to grab our file as it's a bit more intuitive, we do need our location from earlier for this though:

```bash
xrdcp root://dtn-eic.jlab.org:1094//volatile/eic/EPIC//RECO/26.02.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+/DEMPgen-1.2.4_10x250_pi+_q2_3_10_ab.0550.eicrecon.edm4eic.root ./
xrdcp root://epicxrd1.sdcc.bnl.gov:1095//eic/EPIC//RECO/26.03.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+/DEMPgen-1.2.4_10x250_pi+_q2_3_10_ab.0000.eicrecon.edm4eic.root ./
```

Where `./` is our current directory, this time, we'll just get the actual file! We could also specify a new name if desired in place of just `./`.

Our full path, including location, as used here, will also be useful if we want to stream our files directly in a script. In ROOT, we could just do:

```c++
auto f = TFile::Open("root://dtn-eic.jlab.org:1094//volatile/eic/EPIC//RECO/26.02.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+/DEMPgen-1.2.4_10x250_pi+_q2_3_10_ab.0550.eicrecon.edm4eic.root")
auto f = TFile::Open("root://epicxrd1.sdcc.bnl.gov:1095//eic/EPIC//RECO/26.03.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+/DEMPgen-1.2.4_10x250_pi+_q2_3_10_ab.0000.eicrecon.edm4eic.root")
```

or using python and uproot:

```python
import uproot
import XRootD
file_path = "root://dtn-eic.jlab.org:1094//volatile/eic/EPIC//RECO/26.02.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+/DEMPgen-1.2.4_10x250_pi+_q2_3_10_ab.0550.eicrecon.edm4eic.root"
file_path = "root://epicxrd1.sdcc.bnl.gov:1095//eic/EPIC//RECO/26.03.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+/DEMPgen-1.2.4_10x250_pi+_q2_3_10_ab.0000.eicrecon.edm4eic.root"
root_file = uproot.open(file_path)
```

Expand All @@ -409,7 +409,7 @@ or directly with Pyroot:
```python
import ROOT
import XRootD
file_path = "root://dtn-eic.jlab.org:1094//volatile/eic/EPIC//RECO/26.02.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+/DEMPgen-1.2.4_10x250_pi+_q2_3_10_ab.0550.eicrecon.edm4eic.root"
file_path = "root://epicxrd1.sdcc.bnl.gov:1095//eic/EPIC//RECO/26.03.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+/DEMPgen-1.2.4_10x250_pi+_q2_3_10_ab.0000.eicrecon.edm4eic.root"
file = ROOT.TFile.Open(file_path, "READ")
```

Expand All @@ -421,7 +421,7 @@ To test the ROOT approach, we can make a macro called `Test.C` and add:

```c++
void Test(){
auto f = TFile::Open("root://dtn-eic.jlab.org:1094//volatile/eic/EPIC//RECO/26.02.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+/DEMPgen-1.2.4_10x250_pi+_q2_3_10_ab.0550.eicrecon.edm4eic.root");
auto f = TFile::Open("root://epicxrd1.sdcc.bnl.gov:1095//eic/EPIC//RECO/26.03.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+/DEMPgen-1.2.4_10x250_pi+_q2_3_10_ab.0000.eicrecon.edm4eic.root");
auto tree = f->Get<TTree>("events");
Long64_t nEntries = tree->GetEntries(); // read the number of entries in the tree
cout << nEntries << " events in tree" << endl;
Expand All @@ -435,7 +435,7 @@ Similarly in Python, we can make `Test.py` and use Uproot:
```python
import uproot
import XRootD
file_path = "root://dtn-eic.jlab.org:1094//volatile/eic/EPIC//RECO/26.02.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+/DEMPgen-1.2.4_10x250_pi+_q2_3_10_ab.0550.eicrecon.edm4eic.root"
file_path = "root://epicxrd1.sdcc.bnl.gov:1095//eic/EPIC//RECO/26.03.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+/DEMPgen-1.2.4_10x250_pi+_q2_3_10_ab.0000.eicrecon.edm4eic.root"
root_file = uproot.open(file_path)
print(root_file['events'].num_entries, "events in this tree.")
```
Expand All @@ -444,7 +444,7 @@ or directly using PyRoot:
```python
import ROOT
import XRootD
file_path = "root://dtn-eic.jlab.org:1094//volatile/eic/EPIC//RECO/26.02.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+/DEMPgen-1.2.4_10x250_pi+_q2_3_10_ab.0550.eicrecon.edm4eic.root"
file_path = "root://epicxrd1.sdcc.bnl.gov:1095//eic/EPIC//RECO/26.03.0/epic_craterlake/EXCLUSIVE/DEMP/DEMPgen-1.2.4/10x250/q2_3_10/pi+/DEMPgen-1.2.4_10x250_pi+_q2_3_10_ab.0000.eicrecon.edm4eic.root"
file = ROOT.TFile.Open(file_path, "READ")
print((file.Get("events")).GetEntries(), "events in this tree.")
```
Expand Down
20 changes: 10 additions & 10 deletions learners/xrootd.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ In the tutorial, we used XrootD to copy files and to stream files. We could also

Unlike Rucio, if we use XrootD, we *do* have a file structure. Typically, file paths typically look like:

- /volatile/eic/EPIC/<span style="color:#1845fb">OUTPUT_TYPE</span>/<span style="color:#ff5e02">CAMPAGIN</span>/<span style="color:#c91f16">DETECTOR_GEOMETRY</span>/<span style="color:#c849a9">PHYSICS_PROCESS_TYPE</span>/<span style="color:#adad7d">PHYSICS_PROCESS</span>/<span style="color:#86c8dd">BEAM_ENERGY</span>/<span style="color:#578dff">OTHER_CONDITIONS</span>/FILE.eicrecon.edm4eic.root
- /jlab-osdf-ro/eic/EPIC/cache/<span style="color:#1845fb">OUTPUT_TYPE</span>/<span style="color:#ff5e02">CAMPAGIN</span>/<span style="color:#c91f16">DETECTOR_GEOMETRY</span>/<span style="color:#c849a9">PHYSICS_PROCESS_TYPE</span>/<span style="color:#adad7d">PHYSICS_PROCESS</span>/<span style="color:#86c8dd">BEAM_ENERGY</span>/<span style="color:#578dff">OTHER_CONDITIONS</span>/FILE.eicrecon.edm4eic.root
- <span style="color:#1845fb">OUTPUT_TYPE</span> - Is this event generator, simulation or reconstruction output?
- <span style="color:#ff5e02">CAMPAIGN</span> - Which simulation campaign is this from?
- <span style="color:#c91f16">DETECTOR_GEOMETRY</span> - Which detector geometry/design was used to run this simulation?
Expand All @@ -25,7 +25,7 @@ Some directories may include event generator versions after the physics process

## Warning - Non-JLab Servers

Notice the `/volatile/` at the front of the path. This is specific to files on the JLab XrootD server.
Notice the `/jlab-osdf-ro/eic/EPIC/cache/` at the front of the path. This is specific to files on the JLab XrootD server.

Files stored elsewhere may just be under `/eic/EPIC/...`

Expand All @@ -37,21 +37,21 @@ We can browse the simulation output using XrootD from within the eic-shell. To b

```bash
./eic-shell
xrdfs root://dtn-eic.jlab.org
ls /volatile/eic/EPIC/RECO/26.02.0
xrdfs root://dtn2304.jlab.org:8443
ls /jlab-osdf-ro/eic/EPIC/cache/RECO/26.03.0
exit
```

`xrdfs` is the command to log in to a specific server, in this case `root://dtn-eic.jlab.org`.
`xrdfs` is the command to log in to a specific server, in this case `root://dtn2304.jlab.org:8443`.

When calling ls, we should see everything in this subfolder. In this case, all files from the **February 2026** campaign.
When calling ls, we should see everything in this subfolder. In this case, all files from the **March 2026** campaign.

Files can also be copied locally by replacing `ls` with `cp`.

It is also possible to copy a file and open it locally using the `xrdcp` command:
```bash
./eic-shell
xrdcp root://dtn-eic.jlab.org//volatile/eic/EPIC/RECO/26.02.0/path-to-file .
xrdcp root://dtn2304.jlab.org:8443//jlab-osdf-ro/eic/EPIC/cache/RECO/26.03.0/path-to-file .
exit
```

Expand All @@ -67,15 +67,15 @@ In our earlier episode, we used this command to copy a file we found using Rucio
It is also possible to open a file directly in ROOT if you have XrootD installed too. Note that the following command should be executed after opening root and `TFile::Open()` should be used:

```c++
auto f = TFile::Open("root://dtn-eic.jlab.org//volatile/eic/EPIC/RECO/path-to-file")
auto f = TFile::Open("root://dtn2304.jlab.org:8443//jlab-osdf-ro/eic/EPIC/cache/RECO/path-to-file")
```

or using python and uproot:

```python
import uproot
import XRootD
file_path = "root://dtn-eic.jlab.org//volatile/eic/EPIC/RECO/path-to-file"
file_path = "root://dtn2304.jlab.org:8443//jlab-osdf-ro/eic/EPIC/cache/RECO/path-to-file"
root_file = uproot.open(file_path)
```

Expand All @@ -84,6 +84,6 @@ or directly with Pyroot:
```python
import ROOT
import XRootD
file_path = "root://dtn-eic.jlab.org//volatile/eic/EPIC/RECO/path-to-file"
file_path = "root://dtn2304.jlab.org:8443//jlab-osdf-ro/eic/EPIC/cache/RECO/path-to-file"
file = ROOT.TFile.Open(file_path, "READ")
```
Loading