Cloud Data Stores
DDC supports these types of Cloud storages as data stores:
AWS S3 - AWS (Amazon Web Services).
Azure Blobs - Microsoft Azure Blobs (used to store unstructured text and binary data).
Azure Table - lets programs store structured text in partitioned collections of entities that are accessed by partition key and primary key.
Office 365 Sharepoint Online - Sharepoint Online is a document management and storage system delivered as part of Microsoft Online Services suite.
Office 365 Exchange Online - Exchange Online is Exchange Server delivered as a cloud service hosted by Microsoft.
G-Suite (G-Mail and G-Drive)
Note
Before adding any Cloud data store, make sure that you have the required user credentials handy.
Adding Cloud Data Stores
Use the Add Data Store wizard to add a big data type data store. Adding a Big Data data store involves the following steps:
1. Select Store Type
In the Select Store Type screen of the wizard select Cloud in the Select Data Store Category.
From the Select Database Type drop-down list select:
AWS S3
Azure Blobs
Azure Table
Office 365: Sharepoint Online
Office 365: Exchange Online
G-Suite
Click Next to go on to the Configure Connection screen.
2. Configure Connection
In the Configure Connection screen of the wizard, provide the following configuration details for your data store:
AWS S3
Provide the user security credentials, which consist of an Access Key ID and a Secret Access Key.
Access Key ID: Enter the Access Key ID that you obtained from your storage account administrator. For example:
AKIAABCDEFGHIEXAMPLESecret Access Key: Enter the Secret Access Key as obtained from your storage account administrator. For example:
aBcDeFGHiJKLM/A1NOPQR/wxYzdcbAEXAMPLEKEYd
Select the Show Secret Access Key checkbox if you want to view the secret access key.
AZURE BLOBS
In the Configure Connection step, provide the following information:
Account Name: The name of your Azure Storage account.
User: The name of your Azure Storage account.
Active Access Key: Enter key1 or key2, which is your primary or secondary Azure account access key. If you do not know what they are, follow the steps in Obtaining the Azure Account Access Keys.
Tip
You should ask your Azure Storage account administrator which access key is currently active, since only one access key can be active at a time.
AZURE TABLE
Account Name: Enter your Azure account name.
User: Enter your Azure Storage account name.
Password: Your Azure password.
OFFICE 365: SHAREPOINT ONLINE
Domain: Enter your SharePoint Online organization name. For example, if you access SharePoint Online at https://mycompany.sharepoint.com, enter mycompany.
User: Enter a valid SharePoint Online user's email address. The user must have Read permissions to the top-level root site collection, and minimum Read permissions to all site collections, sites and lists to be scanned.
Password: Enter the password for the SharePoint Online user.
OFFICE 365: EXCHANCE ONLINE
Exchange Online Domain: Enter a domain to scan mailboxes that reside on that domain. This is usually the domain component of the email address, or the Windows Domain.
Client ID: Enter your Exchange Online client ID (application ID).
Client Secret Key: Enter your Exchange Online client secret key. Select the Show Client Secret Key check-box to view the key.
Tenant ID: Enter your Office 365: Exchange Online tenant ID. Your Microsoft 365 tenant ID is a globally unique identifier (GUID) that is different than your organization name or domain.
G-SUITE
Domain: The G-Suite domain that you want to scan in the G Suite Domain field. For example, if your G-Suite administrator email is admin@example.com, your G-Suite domain is example.com.
Admin User: The G-Suite administrator account email address. Use the same administrator account used to Enable APIs and Set up Domain-Wide Delegation.
Service Account: Your Service account ID, for example, ddc-service-account@vertical-tuner-322508.iam.gserviceaccount.com.
IP12 Key: Upload the P12 key associated with your Service account ID.
For details, refer to the information in Configuring a G-Suite Account.
The Agent Selection section allows you to specify the minimum and maximum number of proxy agents when adding a datastore. Employing a group of agents instead of a single agent to run the scan should improve the scan execution time.
Note
The multiple agent functionality is not supported for the Office 365 Sharepoint Online datastore.
In the Select Number of Agents menu set the number of agents for the datastore:
Minimum: - Set the minimun number of agents to use to scan the datastore. At least that number of proxy agents must be able to connect to the datastore.
Maximum: - Set the maximum number agents to use to scan the datastore.
Warning
• As there is no limit on the number of minimum and maximum agents that you can set, you should exercise caution so that you do not impact the system performance by using too many resouces for a single scan.
• You will not be able to add a datastore if the minimum number of agents cannot be assigned.
• A scan will fail if the assigned agent is unavailable after adding the datastore.
• The minimum number of agents must be less than or equal to the maximum number of agents.In the Add Label: field, add an agent label, by entering a label or removing and existing label. Agent labels represent the agent capabilities.
Click Next to move on to the General Info step of the wizard.
3. General Info
In the General Info screen of the wizard, specify the name, description, branch location, and sensitivity level for your data store. See "Configuring a Data Store - General Information" for details.
Configure the General Info part per the information in General Info.
Click Next to go to the Add Tags & Access Control screen.
4. Add Tags & Access Control
In the Add Tags & Access Control screen of the wizard, grant access rights to your data store and add metadata. See "Configuring a Data Store – Tags and Access Control" for details.
Configure the Tags & Access Control par per the information in Tags & Access Control.
Click Save. The newly created data store appears on the Data Stores page. By default, data stores are displayed in alphabetic order by name. Depending on the number of entries per page, you might need to navigate to other pages to view the newly created data store.
At any time during the configuration you can click Back to go to any of the previous wizard screens to update the configuration.
The newly created data store appears on the Data Stores page. By default, data stores are displayed in alphabetic order by name. Depending on the number of entries per page, you might need to navigate to other pages to view the newly created data store.
Recommended Least Privilege User Approach: !!! note To reduce the risk of data loss or privileged account abuse, the Target credentials provided for the intended Target should only be granted read-only access to the exact resources and data that require scanning. Never grant full user access privileges or unrestricted data access to any application if it is not required.
Click Save to create the data store. At any time during the configuration you can click Back to go to any of the previous wizard screens to update the configuration.
The newly created data store appears on the Data Stores page. By default, data stores are displayed in alphabetic order by name. Depending on the number of entries per page, you might need to navigate to other pages to view the newly created data store.
Obtaining the Azure Account Access Keys
If you need to find out what your Azure account access keys are:
Log into your Azure account.
Navigate to All resources > [Storage account].
Click Access keys under Settings.
Note down the key1 (primary) and key2 (secondary).
The primary and secondary access keys are used to make rolling key changes. Only one access key can be active at a time. Ask your Azure Storage account administrator which access key is currently active, and use that key to connect DDC to your Azure Storage account.
Configuring a G-Suite Account
Because the Google API imposes certain restrictions on software attempting to access data on their services setting up a G-Suite account as a data store requires more work than other cloud services. You perform the procedure via the Google Cloud Platform console.
Before you can add a G-Suite product as Data Store, you must have a G-Suite administrator account for the target G-Suite domain. It must be a G-Suite account for personal Google accounts are not supported.
To configure your G-Suite account for scanning you need to:
Selecting a Project
Log into the Google Developers Console.
Click the Select a project menu to expand it. The Select dialog box opens and displays a list of existing projects.
In the Select dialog box, you can:
Select an existing project.
Create a new project (recommended).
Select a project in the Google Developers Console to enable G Suite APIs.
To select an existing project:
Click a project.
Click OPEN.
To create a new project:
Click the NEW PROJECT button.
In the New Project page, enter your Project name and click CREATE.
After creating or selecting a project you are taken to its APIs & Services page. In it you can click the +ENABLE APIS AND SERVICES button (at the top of the screen).
Enabling APIs
To enable G-Suite APIs:
Select a Project.
In the project Dashboard, click +ENABLE APIS AND SERVICES. This displays the API Library.
Enable the Admin SDK API.
a. Under G-Suite APIs, click Admin SDK.
b. Click the ENABLE button when the Admin SDK API is displayed. The Admin SDK API statistics screen is displayed.
Repeat the above steps to enable also these APIs:
Gmail API (for Google Mail)
Google Drive API (for Google Drive)
Tip
If you are lost after the previous step, and are not sure where to enable these, click the toolbar icon on the top left side of the Dashboard (the three horizontal dashes), then Home, then Go to APIs overview in the APIs panel (the link at the bottom of it).
Creating a Service Account
To create a service account in the Google Developers Console for use with DDC scans:
Click the menu on the upper-left corner of the Google Developers Console (the three horizontal dashes on the top left side of the Dashboard).
Select IAM & Admin -> Service Accounts.
Click the +CREATE SERVICE ACCOUNT button.
In the Create service account dialog box, enter the following:
a. Service Account Details:
Service account name - Display name for this service account. For example, doc-service-account.
Service account ID - it will be autocompleted with your Service account name (above) and your earlier created postfix. You will need this Service account ID later for configuring DDC. In the Service account for project "such-and-such" you can now see your service account. An example service account ID: ddc-service-account@vertical-tuner-322508.iam.gserviceaccount.com.
Service account description (optional).
b. Click the CREATE AND CONTINUE button.
c. Grant this service account access to project:
- Select a role - click into the text boxt to open a search dialog (Type to filter 'owner', then select 'Owner' Full access to all resources).
d. Click the CONTINUE button.
e. Grant user access to this service account (optional). In this section just click DONE.
To complete the procedure, you need to add a private key. For that:
a. Click the three vertical dots in the Actions column in the Service account for project screen.
b. Then select Manage keys. You will be taken to the Keys screen.
c. Click the ADD KEY pull down menu, and then select the Create new key option.
d. In the Key type, select the P12 radiobutton.
e. Click CREATE. The Service account and key created dialog box displays, and you can save the P12 key to your computer. This key is normally not requiered but you should keep it in a safe location, just in case.
!!! tip The dialog box displays the private key’s password: notasecret - you can disregard this password.f. Click CLOSE.
Write down the newly created service account’s Service account ID and Key ID.
Setting up Domain-Wide Delegation
To be able to access your G-Suite domain with the Service Account, you must set up and enable domain-wide delegation for it. To set up domain-wide delegation for an existing service account:
Click on the service account name that you created in Creating a Service Account step. To use the same example: ddc-service-account@vertical-tuner-322508.iam.gserviceaccount.com.
On the account DETAILS tab, click the SHOW DOMAIN WIDE DELEGATION link at the bottom to expand that section.
Select the Enable Google Workspace Domain-Wide Delegation checkbox. Additional configuration options will be displayed.
Click inside the Product name for the consent screen field and type in the product name. For example: "DDC-service-account".
Click SAVE and a message "Account 'such-and-such' has been updated" is displayed. Also a new field Client ID appears with a client identifier. Note down the client id as you will need it later.
Go to the G-Suite Admin Console. For that, you need to open a new tab or window in your browser.
a. Log in to your G-Suite Admin Console account. It has to be an administrator account.
b. In the navigation panel on the left, click Security -> API controls. Select Security to manage security features in the G Suite admin console.
c. In the API controls screen that is displayed, scroll down to the Domain wide delegation section (at the bottom of the page).
d. In the Domain wide delegation section, click the MANAGE DOMAIN WIDE DELEGATION link. This will take you to a screen displaying a list of all the service accounts that you already have created (the breadcrumb of this screen: Security -> API Controls -> Domain-wide Delegation).
e. Click the Add new link to create a new service account. A pop-up dialog Add a new client ID opens. You need to provide:
Client ID - This is the client ID that you were supposed to note down in the previous step (Step 5).
OAuth Scopes - type in or copy-paste these URIs (i.e. API Scopes). You can use one line and separating them with commas, or use separate fields for each scope:
- Required for all: https://www.googleapis.com/auth/admin.directory.user.readonly
- Google Mail: https://mail.google.com/
- Google Drive: https://www.googleapis.com/auth/drive.readonly
f. Click AUTHORIZE to complete the acccount configuration and a new service account with its scopes is displayed in the list (as "DDC-service-account" to use the same example as in Creating a Service Account).
Navigate back to the API controls screen (click Security -> API controls in the toolbar on the left).
In the App access control panel, click the MANAGE THIRD-PARTY APP ACCESS link to go to a list of Connected apps (breadcrumb: Security -> API controls -> App Access Control).
Click the Configure new app menu and select the option OAuth App Name Or Client ID.
In the Configure an OAuth app screen that opens, search for your client ID. This is the client ID that you were supposed to note down in Step 5. Type it in or paste in the search field and click SEARCH. The search should return your service account name, that is, to use the same example as before, "DDC-service-account".
Click this account in the App name panel and then click the Select button on the right. You will be taken to the app access configuration screen for your client ID.
In the access configuration screen for your client ID:
a. Click to select the checkbox for your client ID.
b. Click SELECT to continue.
c. In the App access screen, you must choose which access type will be applied to your client ID. Select the Trusted radiobutton.
d. Click CONFIGURE to continue.
Now, your service account has "Trusted" in the Access column, in the list of Connected apps and this is what we aimed for.
