How do I change the name of the experiment? I tried to use dvc exp run -n to name the project then use git to push to github. However the experiment name is still SHA.
Tried: I tried to use dvc exp run -n to name the project then use git to push to github. However the experiment name is still SHA.
Expected experiment name on iterative studio interface to display the name
Actually happened: Github SHA value instead of the name
There are two types of experiments in DVC ecosystem that we need to distinguish and there are a few different approaches on naming them.
First, is what we sometimes call "ephemeral" experiments, those that do not create commits in your Git history unless you explicitly say so. They are described in this Get Started section. For each of those experiments a name is auto-generated (e.g. angry-upas in one of the examples from the doc) or you could use dvc exp run -n ... to pass a particular name to it.
Another way to create those experiments (+ send them to Studio) is to use DVC logger (DVCLive), e.g. it's described here. Those experiments will be visible in Studio with an auto-generated names (or a name that was provided when they were created).
Now, we have another type of an experiment - a commit. Something that someone decided to make persistent and share with the team via PR and/or via a regular commit. Those are presented in the image that is shared in the Question.
Since they are regular commits, the regular Git rules apply to them - they have hash, they have descriptions, and most relevant to this discussion is that they have tags. All this information will be reflected in Studio UI.
E.g. in this public example-get-started repo:
It think in your case, tags would be the most natural way to rename those. May be we can introduce a way to push exp name as a git tag along with the experiment when it's being saved. WDYT?
Let me know if that answers your question.
Related
I have an azure blob container with data which I have not uploaded myself. The data is not locally on my computer.
Is it possible to use dvc to download the data to my computer when I haven’t uploaded the data with dvc? Is it possible with dvc import-url?
I have tried using dvc pull, but can only get it to work if I already have the data locally on the computer and have used dvc add and dvc push .
And if I do it that way, then the folders on azure are not human-readable. Is it possible to upload them in a human-readable format?
If it is not possible is there then another way to download data automatically from azure?
I'll build up on #Shcheklein's great answer - specifically on the 'external dependencies' proposal - and focus on your last question, i.e. "another way to download data automatically from Azure".
Assumptions
Let's assume the following:
We're using a DVC pipeline, specified in an existing dvc.yaml file. The first stage in the current pipeline is called prepare.
Our data is stored on some Azure blob storage container, in a folder named dataset/. This folder follows a structure of sub-folders that we'd like to keep intact.
The Azure blob storage container has been configured in our DVC environment as a DVC 'data remote', with name myazure (more info about DVC 'data remotes' here)
High-level idea
One possibility is to start the DVC pipeline by synchronizing a local dataset/ folder with the dataset/ folder on the remote container.
This can be achieved with a command-line tool called azcopy, which is available for Windows, Linux and macOS.
As recommended here, it is a good idea to add azcopy to your account or system path, so that you can call this application from any directory on your system.
The high-level idea is:
Add an initial update_dataset stage to the DVC pipeline that checks if changes have been made in the remote dataset/ directory (i.e., file additions, modifications or removals).
If changes are detected, the update_datset stage shall use the azcopy sync [src] [dst] command to apply the changes on the Azure blob storage container (the [src]) to the local dataset/ folder (the [dst])
Add a dependency between update_dataset and the subsequent DVC pipeline stage prepare, using a 'dummy' file. This file should be added to (a) the outputs of the update_dataset stage; and (b) the dependencies of the prepare stage.
Implementation
This procedure has been tested on Windows 10.
Add a simple update_dataset stage to the DVC pipeline by running:
$ dvc stage add -n update_dataset -d remote://myazure/dataset/ -o .dataset_updated azcopy sync \"https://[account].blob.core.windows.net/[container]/dataset?[sas token]\" \"dataset/\" --delete-destination=\"true\"
Notice how we specify the 'dummy' file .dataset_updated as an output of the stage.
Edit the dvc.yaml file directly to modify the command of the update_dataset stage. After the modifications, the command shall (a) create the .dataset_updated file after the azcopy command - touch .dataset_updated - and (b) pass the current date and time to the .dataset_updated file to guarantee uniqueness between different update events - echo %date%-%time% > .dataset_updated.
stages:
update_dataset:
cmd: azcopy sync "https://[account].blob.core.windows.net/[container]/dataset?[sas token]" "dataset/" --delete-destination="true" && touch .dataset_updated && echo %date%-%time% > .dataset_updated # updated command
deps:
- remote://myazure/dataset/
outs:
- .dataset_updated
...
I recommend editing the dvc.yaml file directly to modify the command, as I wasn't able to come up with a complete dvc add stage command that took care of everything in one go.
This is due to the use of multiple commands chained by &&, special characters in the Azure connection string, and the echo expression that needs to be evaluated dynamically.
To make the prepare stage depend on the .dataset_updated file, edit the dvc.yaml file directly to add the new dependency, e.g.:
stages:
prepare:
cmd: <some command>
deps:
- .dataset_updated # add new dependency here
- ... # all other dependencies
...
Finally, you can test different scenarios on your remote side - e.g., adding, modifying or deleting files - and check what happens when you run the DVC pipeline up till the prepare stage:
$ dvc repro prepare
Notes
The solution presented above is very similar to the example given in DVC's external dependencies documentation.
Instead of the az copy command, it uses azcopy sync.
The advantage of azcopy sync is that it only applies the differences between your local and remote folders, instead of 'blindly' downloading everything from the remote side when differences are detected.
This example relies on a full connection string with an SAS token, but you can probably do without it if you configure azcopy with your credentials or fetch the appropriate values from environment variables
When defining the DVC pipeline stage, I've intentionally left out an output dependency with the local dataset/ folder - i.e. the -o dataset part - as it was causing the azcopy command to fail. I think this is because DVC automatically clears the folders specified as output dependencies when you reproduce a stage.
When defining the azcopy command, I've included the --delete-destination="true" option. This allows synchronization of deleted files, i.e. files are deleted on your local dataset folder if deleted on the Azure container.
Please, bear with me, since you have a lot of questions. Answer needs a bit structure and background to be useful. Or skip to the very end to find some new ways of doing Is it possible to upload them in a human-readable format? :). Anyways, please let me know if that solves your problem, and in general would be great to have a better description of what you are trying to accomplish at the end (high level description).
You are right that by default DVC structures its remote in a content-addressable way (which makes it non human-readable). There are pros and cons to this. It's easy to deduplicate data, it's easy to enforce immutability and make sure that no one can touch it directly and remove something, directory names in projects make it connected to actual project and their meaning, etc.
Some materials on this: Versioning Data and Models, my answer of on how DVC structures its data, upcoming Data Management User Guide section (WIP still).
Saying that, it's clear there are downsides to this approach, especially when it comes to managing a lot of objects in the cloud (e.g. millions of images, etc). To name a few concerns that I see a lot as a pattern:
Data has been created (and being updated) by someone else. There is some ETL, third party tool, etc. We need to keep that format.
Third party tool expect to have data in "human" readable way. It doesn't integrate with DVC to being able to access it indirectly via Git. (one of the examples - Label Studio need direct links to S3).
It's not practical to move all of data into DVC, it doesn't make sense to instantiate all the files at once as one directory. Users need slices, usually based on some annotations (metadata), etc.
So, DVC has multiple features to deal with data in its own original layout:
dvc import-url - it'll download objects, it'll cache them, and will by default push (dvc push) to remote to again save them to guarantee reproducibility (this can be changed). This command creates a special file .dvc that is being used to detect changes in the cloud to see if DVC needs to download something again. It should cover the case for "to download data automatically from azure".
dvc get-url - this more or less wget or rclone or aws s3 cp, etc with multi cloud support. It just downloads objects.
A bit advanced thing (if you DVC pipelines):
Similar to import-url but for DVC pipelines - external dependencies
The the third (new) option. It's in beta phase, it's called "cloud versioning" and essentially it tries to keep the storage human readable while still benefit from using .dvc files in Git if you need them to reference an exact version of the data.
Cloud Versioning with DVC (it's WPI when I write this, if PR is merged it means you can find it in the docs
The document summarizes well the approach:
DVC supports the use of cloud object versioning for cases where users prefer to retain their original filenames and directory hierarchy in remote storage, in exchange for losing the de-duplication and performance benefits of content-addressable storage. When cloud versioning is enabled, DVC will store files in the remote according to their original directory location and
filenames. Different versions of a file will then be stored as separate versions of the corresponding object in cloud storage.
I have some Windows Server 2016 instances on GCE (for Jenkins agents).
I'm wondering what is the best/good practice when it comes to computer name.
Currently, when I want to create a new node, I clone an instance (create images from disks + create template + create instance from template).
On this clone, I change the computer name (in Windows) so that it has the same name as on GCE. Is it useful? recommended? bad? needed?
I know that the name of the Jenkins node needs to be the same as the name of the GCE instance (to be picked up easily). However, I don't think the Windows computer name matters.
So, should I pick an identical generic name for all of them? A prefix+random generated name? Continue with the instance=computer=node name?
The node name that I use in Jenkins is always retrieved from env.NODE_NAME (when needed), so that should not break any pipeline. Not sure thought, as I may be missing something (internal to Jenkins).
Bonus question: After cloning, I have to do some modifications on the clone for Perforce (p4) to work.
I temporarily set some env variables
I duplicate the workspace: p4 client -t prefix-buildX-suffix prefix-buildY-suffix
I setup the stream (not sure if doable in one step)
Then regenerate the list of files: p4 sync -k <root_folder_to_be_generated>/...#YYYY/MM/DD
So, here also there's a name prefix-buildY-suffix which is the same as the one from the instance=computer=node (buildY). It may be a separate question, but as it's still from the same context, I'm putting it here: should I recreate a new workspace all the time? Knowing that it's on several machines, I'd say yes. Otherwise, I "imagine" that p4 would have contradictory information about the state of this workspace. So, here also, I currently need to customize the name. So, even if I make the Windows computer name generic, I would still need to customize the p4 workspace name, wouldn't I?
Jenkins must have the same computer name as the one on the network.
So, all three names must be identical.
Problem:
I've got a manual intervention step with textual steps for the person performing the deployment to follow.
I'd like to pass in the name of the target server so the the person doesn't need to lookup the server name being targeted.
For example as you see below, I need them to unzip to a location on the target server.
**SECTION 1: (Main installation)**
1. Navigate to: #{InstallationZipLocation}.
2. Download zip file named: #{ZipFileName}
3. Unzip to the desktop on: #{DeploymentTargetMachineName} --need help here
4. Run executable named: #{ExecutableName}
5. Accept default settings
What I have tried:
Octopus Deploy - System Variables Documentation offers:
#{Octopus.Deployment.Machines} results in: Machines-6
#{Octopus.Deployment.SpecificMachines} results in: (empty string)
What I expect to see:
3. Unzip to the desktop on: FTPServer05
Additional Comment:
I realize I could set the name of the target server in my variables list for each target environment/scope, resulting in only 4 variables (not a big deal, and easy to maintain), but I was curious if there was a way to simplify it. We are running Octopus Deploy 3.12.9.
So I was looking for an easier approach, but stumbled on something that I found to be rather interesting so I went ahead and implemented it.
Output variables. . . "After a step runs, Octopus captures the output variables, and keeps them for use in subsequent steps."
What I did to resolve my issue:
I setup a custom step-template which sole purpose is to set "output variables" to use in my subsequent step. You could have this be your first step in your project, or at a minimum come before the step that references the variable you are setting.
Custom step setup:
Powershell:
Write-Host "TargetDeploymentMachineName $TargetDeploymentMachineName"
Set-OctopusVariable -name "TargetDeploymentMachineName" -value $TargetDeploymentMachineName
Parameters:
Then in my Manual Intervention step, I use the output value like so:
3. Unzip to the desktop on: #{Octopus.Action[MyProject-Set-Output-Variables].Output.TargetDeploymentMachineName}
(Where [MyProject-Set-Output-Variables] represents the name of the step in my deployment project which is responsible for assigning the output variables)
Explanation for why I was having trouble in my question:
Turns out the variable binding syntax to use for my question would have been:
Octopus.Machine.Name = The name that was used to register the
machine in Octopus. Not the same as Hostname
However, the Manual Intervention step specifically does not have a "Deployment Target":
It instead just runs on "Octopus Server":
So I am pretty sure that is why I was not getting a value for the "target". For example, I simply tested another new basic step that used the "Deployment Targets" radio buttons, which resulted in the FTPServer05 value I was expecting.
I built an Analysis that displayed Results, error free. All is well.
Then, I added some filters to existing criteria sets. I also copied an existing criteria set, pasted it, and modified it's filters. When I try to display results, I see a View Display Error.
I’d like to revert back to that earlier functional version of the analyses, hopefully without manually undoing the all of filter & criteria changes I made since then.
If you’ve seen a feature like this, I’d like to hear about it!
Micah-
Great question. There are many times in the past when we wished we had some simple SCM on the Oracle BI Web Catalog. There is currently no "out of the box" source control for the web catalog, but some simple work-arounds do exist.
If you have access server side where the web catalog lives you can start with the following approach.
Oracle BI Web Catalog Version Control Using GIT Server Side with CRON Job:
Make a backup of your web catalog!
Create a GIT Repository in the web cat
base directory where the root dir and root.atr file exist.
Initial commit eveything. ( git add -A; git commit -a -m
"initial commit"; git push )
Setup a CRON job to run a script Hourly,
Minutely, etc that will tell GIT to auto commit any
adds/deletes/modifications to your GIT repository. ( git add -A; git
commit -a -m "auto_commit_$(date +"%Y-%m-%d_%T")"; git push )
Here are the issues with this approach:
If the CRON runs hourly, and an Analysis changes 3 times in the hour
you'll be missing some versions in there.
No actual user submitted commit messages.
Object details such as the Objects pretty "Name" (caption), Description (user populated on Save Dialog), ACLs, and object custom properties are stored in a binary file format. These files have the .atr extension. The good news though is that the actual object definition is stored in a plain text file in XML (Without the .atr).
Take this as a baseline, and build upon it. Here is how you could step it up!
Use incron or other inotify based file monitoring such as ruby
based guard. Using this approach you could commit nearly
instantly anytime a user saves an object and the BI server updates
the file system.
Along with inotify, you could leverage the BI Soap API to retrieve the actual object details such as Description. This would allow you to create meaningfull commit messages. Or, parse the binary .atr file and pull the info out. Here are some good links to learn more about Web Cat ATR files: Link (Keep in mind this links are discussing OBI 10g. The binary format for 11G has changed slightly.)
When setting up a new Hudson/Jenkins instance i run into the problem that i have to manually provide all the email addresses for the scm users.
We are using subversion and i can't generate the mail addresses from the usernames. I got a mapping but i found no way to copy / edit that without making use of the gui. With 20+ users that gets boring and i'd like to have just edit a file or something.
Maybe i'm missing some trivial thing like a scmusers.xml (which totally would do the job) ?
I've got 2 solutions so far:
The users are stored in users/USERNAME/config.xml could be versioned / updated / etc.
Makeing use of the RegEx+Email+Plugin, create one rule per user and version that file.
With 20+ users, setting up a list for the scm users is the way to go. Then when folks add/leave the group, you only have to edit the mailing list instead of the Hudson jobs. Also depending on your mailing list software, folks might be able to add and drop themselves from the list which would save you the time of maintaining it yourself in Hudson.
You might also want to look into the alias support of whatever email server your Hudson server is using. Let Hudson send out the emails it wants to using the SVN usernames, but then define aliases in your /etc/aliases file (or equivalent for your email server) that map the SVN usernames to the actual email addresses.