Storage research data (Faculty of Science): Difference between revisions

From ICT science
Jump to navigation Jump to search
(Created page with "==Betastor large volume research data storage== '''Address:''' ''betastor.science.uu.nl'' '''Protocols:''' CIFS '''How to mount on your computer:''' \\betastor.science.uu.nl...")
 
 
(124 intermediate revisions by the same user not shown)
Line 1: Line 1:
==Betastor large volume research data storage==
Large volume group storage facility. This documentation replaces the Bètastor documentation for (migrated) storage projects after 22-05-2018. For other large volume research data storage solutions also see [https://www.uu.nl/en/research/research-data-management/it-solutions IT-solutions matrix] for university services (RDM-website)  
'''Address:''' ''betastor.science.uu.nl''
'''Protocols:''' CIFS
'''How to mount on your computer:''' \\betastor.science.uu.nl\ (for Windows OS), cifs://betastor.science.uu.nl/ (for Mac OS).


More detailed instructions can be found under [[Project_space#How_to_connect_to_a_project_space.3F|How_to_connect_to_a_project_space]] replace address with betastor.science.uu.nl
__TOC__
==Storage characteristics==
Project storage is a UU (ITS) facilitated network storage with a functional layer provided by the Science Faculty (ICT-Beta). This means in-house developed and maintained network storage located in the Netherlands (Almere datacenter  and UMC-Science Park).
For data policy guidelines see: [[Storage_research_data_(general)]]
===data types===
* suitable for privacy sensitive data only when the data is encrypted (no default provision), always consult [https://www.uu.nl/en/research/research-data-management/what-we-do/consultancy-and-advice RDM-support] when in doubt
* suitable for large volume research data, sharing data with UU-employees/students only (solisID required)


* How to manage Users and roles with the  [[Self_service_BSSP|Beta Self Service Provider]], Faculty Storage only
===functionality===
* choice between single (NetApp RAID triple parity) or double storage (disaster recovery)
* default 24 x hourly data snapshots current day + 1 daily snapshot kept for 60 days (snapshots use up additional space depending on amount of file changes)
* auto compression and de-duplication of data (resulting in lower data use depending on file type)
* usage can be maximized by quota system: possibility to set folder quota
* self service user management provided ([[Project_storage_Faculty_of_Science#How_to_manage_Users_and_roles|see below]])
* available protocols: smb/cifs, sftp, default mount on Gemini HPC (Linux interactive shell)
 
===price===
* cost 2 euro/TB/month for single storage or 4 euro for double storage (aka backup or disaster recovery)
* costs are charged on basis of the reserved volume size (expandable at any time)
 
===service level===
weekdays from 8 to 17:30, contact the [https://topdesk.ictsc.uu.nl/tas/public/login/form servicedesk]
 
==Request storage==
A request can be placed through this [https://intranet.science.uu.nl/ICTintake form]. In an intake we will determine the most appropriate solution. What is needed to complete the request:
:* name of the Principle Investigator or project manager
:* project name
:* in case of website: url friendly web name
:* WBS of the project (this number will be billed)
:* initial Volume size (in TB's, can be enlarged at any time)
:* name of a data contact (user management)
:* solis-id's of the initial user group
 
==Getting started==
===How to mount the network drive on your computer:===
*: '''''Use the OS finder to connect, not an internet browser!'''''
 
*: Replace ''projectname'' with your own project or leave out (you will be presented with a list)
 
*: Login using: soliscom\[solisID] and [solispassword]
 
<code>Windows OS: \\soliscom.uu.nl\science\projects\<projectname>
Mac OS: smb://soliscom.uu.nl/science/projects/<projectname></code>
 
Details can be found on the [[Project_storage:_how_to]] page
 
===Structure of a project ===
'''What do you see when mounting your project?'''
 
By default you will see just that: a blank project space that can be structured by creating (sub)folders as you need them.
 
'''Functional folders that can be added to a project on request'''
:* ''a personal userfolder'' (personal storage for each authorised user in a project). These are shared ''read only'' within a project. Subfolder ''private'' can be added if sharing is undesirable
:*  an ''archive folder''. This folder typically contains immutable historical data and is readonly for project users (can be configured otherwise by data-manager)
:*  a ''project folder''. This folder is open to all project users to share data between all users (the same as a default blank project, then create your own sub-folders)
:* a ''subproject folder''. This creates a folder with its own user management (to facilitate a restricted group within a project).
 
Contact your groups data-manager to request functional folders
 
===How to manage Users and roles===
See [[Self_service_BSSP|Beta Self Service Provider]] for user-access management, ''datamanagers only''. A ''user'' can be anyone with a solis-id; both students and employees.
 
==Additional services==
 
===Project website===
<code>Data: \\soliscom.uu.nl\science\wwwprojects\<projectname>
Website: <nowiki>https://www.projects.science.uu.nl/<projectname></nowiki></code>
 
For more info see: [[Project_storage:_how_to]]
 
===Measurement service===
When collecting data from imagers  or other local lab systems, usually a lot of manual copying of the resulting data files is involved. We can help you set up a service that will do the copying for you. It can also make sure the lab system doesn't fill up with old data runs. This has to be tailored to your situation. Inquiries may be sent to: [mailto:r.vandenbroek@uu.nl Rob van den Broek]
 
Example of a measurement service (before and after): [[Media:MeasurementService.pdf|diagram]]
 
==Also see==
* [[Personal storage]]
* [[Web sites on faculty servers]]: information on available web platforms
* [[Personal web site]]: technical details when using a web interface on your project space
 
[[Category:Index]]
[[Category:Facilities]]
[[Category:Research and ICT]]
[[Category:Personal ICT]]
 
<!--
==Setup==
* '''Address:''' ''betastor.science.uu.nl''
* '''Protocols:'''
:- CIFS ''standard'' (you need [[VPN|VPN]] outside the UU network)
:- Webdav ''on request'' (no VPN needed outside the UU network, but not well suited for large data files. We recommend you to use CIFS if possible)
===How to mount the network drive on your computer:===
'''Windows OS:''' \\betastor.science.uu.nl\projectname\
  '''Mac OS:''' cifs://betastor.science.uu.nl/projectname
 
If you don't know your exact projectname, you can leave it out. You will be presented with a directory listing of all projects. It is recommended to map your network drive to a fixed drive-letter for future use (Windows) or mark the network address as a favourite (Mac OS).
These instructions can be found under [[Project_space#How_to_connect_to_a_project_space.3F|How_to_connect_to_a_project_space]] replace address with \\betastor.science.uu.nl\projectname\
===view===
Depending on the way your project is set up you will see:
: - your personal user folder (read/write/delete) or all user folders (read only, optional)
: - a project folder (read/write/delete) intended to exchange data within project
: - an archive folder (read or read/write, optional) intended to consolidate finished research(data)
 
===How to synchronise your local data to b&egrave;tastor===
'''Setting up an automated sync'''
 
You can of course copy your local data to the network storage by hand, but this becomes cumbersome after a while and prone to mistakes. To set up an automated sync from Windows or Mac OS read:
[[Storage_synchronize_data]]
 
'''Prevent Mac OS from .DS_Store file creation on network volumes'''
 
Prevent OS X from creating new .DS_Store files when opening folders on remote volumes. The creation of .DS_Store files is frequently the source of complaints against Mac users, who often leave a trail of these files scattered throughout the filesystem when "visiting" a Windows computer. Even with this fix in place, the .DS_Store files will continue to be created on local volumes (which is a good thing).
 
To prevent the creation of these files, open the Terminal and type:
<pre>defaults write com.apple.desktopservices DSDontWriteNetworkStores true</pre>
It may be necessary to log out and back in, or even to restart the computer, for the change to take effect.
 
===How to restore previous versions of a file (Windows only)===
You can use previous versions to restore files and folders that you accidentally modified or deleted, or that were damaged. Depending on the type of file or folder, you can open, save to a different location, or restore a previous version. ''Snapshots are not kept indefinitely. 10% of storage space is reserved for previous versions, so the desired version might not be available depending on time passed and total read/write activity''
 
*''How do I restore previous versions of a file or folder?''
Right-click the file or folder, and then click Restore previous versions.
You’ll see a list of available previous versions of the file or folder.
 
Before restoring a previous version of a file or folder, select the previous version, and then click Open to view it to make sure it’s the version you want.
 
NOTE: You can’t open or copy previous versions of files that were created by Windows Backup, but you can restore them.
 
To restore a previous version, select the previous version, and then click Restore.
 
WARNING: The file or folder will replace the current version on your computer, and the replacement cannot be undone.
 
NOTE: If the Restore button isn’t available, you can’t restore a previous version of the file or folder to its original location. However, you might be able to open it or save it to a different location.
 
*''Can I restore a file or folder that was deleted or renamed?''
If you accidentally delete or rename a file or folder, you can restore a previous version of that file or folder, but you need to know the location where the file or folder was saved. To restore a file or folder that was deleted or renamed, follow these steps:
 
:- Click the Computer icon on your desktop to open it up. Navigate to the folder that used to contain the file or folder, right-click it, and then click Restore previous versions. If the folder was at the top level of a drive, for example H:\, right-click the drive and then click Restore previous versions.
:- Double-click a previous version of the folder that contains the file or folder you want to restore. (For example, if a file was deleted today, choose a version of the folder from yesterday that contains the file.)
:- Drag the file or folder that you want to restore to another location, such as your desktop or another folder.
 
The version of the file or folder is saved to the location that you selected.
 
''If you cannot find the deleted file or you have to restore an entire folder, it is best to start a support call in the [https://topdesk.ictsc.uu.nl/tas/public/login/form UU-servicedesk]''
 
===How to manage Users and roles===
* [[Self_service_BSSP|Beta Self Service Provider]], ''datamanagers only''. A ''user'' can be anyone with a solis-id; both students and employees.
What can be managed using self service?
:# add or revoke user access to a ''personal storage folder'' in the project storage (solis-id only): ''BSSP group: List science_ bstor_ projectName- BSSP''
:# add or revoke user access to the project storage '''archive''': ''BSSP group: List science_ bstor_ projectName_ archief- BSSP (read only) or List science_ bstor_ projectName_ archief-R_BSSP (read/write)''
:# ad or revoke read only user acces to ''all other user folders'' in the project storage: ''BSSP group: List science_ bstor_ projectName-R_BSSP''
:# add a ''project folder'' as a shared space for all participants: ''contact data support (applies to all project members, no user management)''
[[image:FilefolderNWD.png]]
 
==Storage characteristics==
B&egrave;tastor is UU (ITS) facilitated network storage with a functional layer provided by the Science Faculty (ICT-Beta). This means in-house developed and maintained Dutch based network storage currently in one datacenter, but a second location is in development.
===data types===
* suitable for privacy sensitive data when the data is encrypted (no default provision), always consult [https://www.uu.nl/en/research/research-data-management/what-we-do/consultancy-and-advice RDM-support] when in doubt
* suitable for large volume research data, no special facilities for archiving, preserving or sharing data
 
===functionality===
* single storage: NetApp RAID triple parity (back-up location is in development)
* 2x daily snap-shots. This is a dynamic system depending on the storage activity. Normally: going back at least 2 weeks should be possible
* auto compression and de-duplication of data (resulting in lower data use depending on file type)
* usage can be maximized by quota system: possibility to set folder quota
* self service user management provided (see above)
 
===price===
* cost ranging form 2,20 euro/TB/month for single storage (including snapshot storage) to 4,40 with backup storage (in development)
* costs are charged on basis of the reserved volume (expandable at any time)
 
===request storage===
A request can be placed through the [https://topdesk.ictsc.uu.nl/tas/public/login/form servicedesk] or [mailto:s.holleman@uu.nl faculty data contact]. In an intake we will determine the most appropriate solution. What is needed to complete the request:
:* name of the PI or project manger
:* WBS of the project (this number will be billed)
:* initial Volume size (in TB's, can be enlarged at any time)
:* name of a data contact (user management)
:* solis-id's of the initial user group
 
===service level===
weekdays from 8 to 17:30, contact the [https://topdesk.ictsc.uu.nl/tas/public/login/form servicedesk]
 
===Project storage delivered===
:- Experimental Pharmacology
:- FI-TLL, FI-Researchers, FI-Students
 
[[Category:Index]]
[[Category:Facilities]]
[[Category:Voorzieningen]]
[[Category:Research and ICT]]
-->

Latest revision as of 13:43, 26 January 2023

Large volume group storage facility. This documentation replaces the Bètastor documentation for (migrated) storage projects after 22-05-2018. For other large volume research data storage solutions also see IT-solutions matrix for university services (RDM-website)

Storage characteristics

Project storage is a UU (ITS) facilitated network storage with a functional layer provided by the Science Faculty (ICT-Beta). This means in-house developed and maintained network storage located in the Netherlands (Almere datacenter and UMC-Science Park). For data policy guidelines see: Storage_research_data_(general)

data types

  • suitable for privacy sensitive data only when the data is encrypted (no default provision), always consult RDM-support when in doubt
  • suitable for large volume research data, sharing data with UU-employees/students only (solisID required)

functionality

  • choice between single (NetApp RAID triple parity) or double storage (disaster recovery)
  • default 24 x hourly data snapshots current day + 1 daily snapshot kept for 60 days (snapshots use up additional space depending on amount of file changes)
  • auto compression and de-duplication of data (resulting in lower data use depending on file type)
  • usage can be maximized by quota system: possibility to set folder quota
  • self service user management provided (see below)
  • available protocols: smb/cifs, sftp, default mount on Gemini HPC (Linux interactive shell)

price

  • cost 2 euro/TB/month for single storage or 4 euro for double storage (aka backup or disaster recovery)
  • costs are charged on basis of the reserved volume size (expandable at any time)

service level

weekdays from 8 to 17:30, contact the servicedesk

Request storage

A request can be placed through this form. In an intake we will determine the most appropriate solution. What is needed to complete the request:

  • name of the Principle Investigator or project manager
  • project name
  • in case of website: url friendly web name
  • WBS of the project (this number will be billed)
  • initial Volume size (in TB's, can be enlarged at any time)
  • name of a data contact (user management)
  • solis-id's of the initial user group

Getting started

How to mount the network drive on your computer:

  • Use the OS finder to connect, not an internet browser!
  • Replace projectname with your own project or leave out (you will be presented with a list)
  • Login using: soliscom\[solisID] and [solispassword]

Windows OS: \\soliscom.uu.nl\science\projects\<projectname> Mac OS: smb://soliscom.uu.nl/science/projects/<projectname>

Details can be found on the Project_storage:_how_to page

Structure of a project

What do you see when mounting your project?

By default you will see just that: a blank project space that can be structured by creating (sub)folders as you need them.

Functional folders that can be added to a project on request

  • a personal userfolder (personal storage for each authorised user in a project). These are shared read only within a project. Subfolder private can be added if sharing is undesirable
  • an archive folder. This folder typically contains immutable historical data and is readonly for project users (can be configured otherwise by data-manager)
  • a project folder. This folder is open to all project users to share data between all users (the same as a default blank project, then create your own sub-folders)
  • a subproject folder. This creates a folder with its own user management (to facilitate a restricted group within a project).

Contact your groups data-manager to request functional folders

How to manage Users and roles

See Beta Self Service Provider for user-access management, datamanagers only. A user can be anyone with a solis-id; both students and employees.

Additional services

Project website

Data: \\soliscom.uu.nl\science\wwwprojects\<projectname> Website: https://www.projects.science.uu.nl/<projectname>

For more info see: Project_storage:_how_to

Measurement service

When collecting data from imagers or other local lab systems, usually a lot of manual copying of the resulting data files is involved. We can help you set up a service that will do the copying for you. It can also make sure the lab system doesn't fill up with old data runs. This has to be tailored to your situation. Inquiries may be sent to: Rob van den Broek

Example of a measurement service (before and after): diagram

Also see


Retrieved from "http://ict.science.uu.nl/index.php?title=Storage_research_data_(Faculty_of_Science)&oldid=9645"