Skip to content

feat: LSDV-4974: Add status and debug info to storages #4059

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 65 commits into from
May 17, 2023
Merged
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
86baec9
docs: LSDV-4775: Small fixes in storage docs
makseq Apr 15, 2023
9fcb3fe
Revert
makseq Apr 15, 2023
9c1901e
feat: LDSV-4979: Add status and debug info to storages
makseq Apr 18, 2023
b54218a
Fix ldclient debug level
makseq Apr 18, 2023
f7c99d7
Merge branch 'develop' of github.com:heartexlabs/label-studio into fb…
makseq Apr 18, 2023
9305323
Merge branch 'develop' of github.com:heartexlabs/label-studio into fb…
makseq Apr 18, 2023
55f608a
Fixes
makseq Apr 19, 2023
a66fe40
Fixes with storages
makseq Apr 20, 2023
dad71fe
Fix NGINX downloads for export
makseq Apr 20, 2023
7720447
Add migrations
makseq Apr 20, 2023
98a1d52
Fix bug with export annotations and expand_fields in BaseExportSerial…
makseq Apr 21, 2023
3fe0fea
More feats
makseq Apr 22, 2023
54cc233
Refactoring
makseq Apr 23, 2023
9d0792d
Update readme
makseq Apr 24, 2023
d07eed4
Add mermaid
makseq Apr 24, 2023
07ef426
Add mermaid
makseq Apr 24, 2023
6ed0cdc
Some
makseq Apr 24, 2023
567be7c
Merge branch 'fb-lsdv-4979' of github.com:heartexlabs/label-studio in…
makseq Apr 24, 2023
bc71115
Fix last ping
makseq Apr 24, 2023
f81022c
Fix duration
makseq Apr 24, 2023
c6c27ce
Add validation on export storage add sync
makseq Apr 24, 2023
616f44b
Add double validation in API
makseq Apr 24, 2023
f9f5759
Fix validation api
makseq Apr 24, 2023
b30d10c
Rename job_status
makseq Apr 24, 2023
ea4abc0
Fixes
makseq Apr 24, 2023
9f723e5
Fix readme
makseq Apr 24, 2023
340a40f
Add more info for statuses in readme
makseq Apr 24, 2023
a14b560
Fix readme
makseq Apr 24, 2023
0a440b6
Add icon
makseq Apr 24, 2023
bfd8213
Add more help
makseq Apr 24, 2023
9814429
Fix texts
makseq Apr 24, 2023
c788e3e
Some fixes
makseq May 5, 2023
8a3c1c6
Fixes in storage validation
makseq May 5, 2023
54c5a8a
Merge branch 'develop' of github.com:heartexlabs/label-studio into fb…
makseq May 10, 2023
4ccde77
ci: Build frontend
robot-ci-heartex May 10, 2023
7df9ac3
Fix tests in LSE
makseq May 10, 2023
1c05ecf
Merge branch 'fb-lsdv-4979' of github.com:heartexlabs/label-studio in…
makseq May 10, 2023
cb49fd5
Fixes for azure export storage validation
makseq May 10, 2023
27e46aa
Fixed tests and added getLastTraceback
makseq May 11, 2023
4eb091c
ci: Build frontend
robot-ci-heartex May 11, 2023
343b3da
Fixes
makseq May 11, 2023
514352f
Merge branch 'fb-lsdv-4979' of github.com:heartexlabs/label-studio in…
makseq May 11, 2023
b84d8ed
Merge branch 'develop' of github.com:heartexlabs/label-studio into fb…
makseq May 11, 2023
b795709
Fix pytest
makseq May 11, 2023
5597196
Add migration
makseq May 11, 2023
5960fe9
Merge branch 'develop' of github.com:heartexlabs/label-studio into fb…
makseq May 11, 2023
06c861f
Tests
makseq May 12, 2023
368e11a
Fix wrong test
makseq May 12, 2023
df23499
Fix update progross in the correct place
makseq May 12, 2023
2ea899f
Fixes with time_in_progress
makseq May 12, 2023
cc787d5
Fix frontend
makseq May 12, 2023
9f3493c
ci: Build frontend
robot-ci-heartex May 12, 2023
3d8fb11
Fixes
makseq May 12, 2023
4adda8d
ci: Build frontend
robot-ci-heartex May 12, 2023
1bade66
fix: Presigned storage tests (#4184)
bmartel May 12, 2023
0cbb3f3
Fix azure validation
makseq May 13, 2023
f6a9813
Update LSP
makseq May 14, 2023
2e2ee03
ci: Build frontend
robot-ci-heartex May 14, 2023
61b195e
Remove wrong placeholder in azure
makseq May 14, 2023
d85fec2
Add migrations
makseq May 14, 2023
0030b69
Fix null in total annotations in LSP
makseq May 14, 2023
471c040
Update LSP build
makseq May 14, 2023
b7f1fa3
Merge branch 'fb-lsdv-4979' of github.com:heartexlabs/label-studio in…
makseq May 14, 2023
8e3b6c2
Use timezone.now() instead of datetime
makseq May 14, 2023
a8a7e05
Fix migrations
makseq May 14, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion label_studio/core/settings/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@
},
'ldclient.util': {
'handlers': ['console'],
'level': 'WARNING',
'level': 'ERROR',
},
},
}
Expand Down Expand Up @@ -587,6 +587,9 @@ def collect_versions_dummy(**kwargs):

FUTURE_SAVE_TASK_TO_STORAGE = get_bool_env('FUTURE_SAVE_TASK_TO_STORAGE', default=False)
FUTURE_SAVE_TASK_TO_STORAGE_JSON_EXT = get_bool_env('FUTURE_SAVE_TASK_TO_STORAGE_JSON_EXT', default=True)
STORAGE_IN_PROGRESS_TIMER = get_env('STORAGE_IN_PROGRESS_TIMER', 10.0)

USE_NGINX_FOR_EXPORT_DOWNLOADS = get_bool_env('USE_NGINX_FOR_EXPORT_DOWNLOADS', False)

if get_env('MINIO_STORAGE_ENDPOINT') and not get_bool_env('MINIO_SKIP', False):
CLOUD_FILE_STORAGE_ENABLED = True
Expand Down
26 changes: 18 additions & 8 deletions label_studio/data_export/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -469,14 +469,24 @@ def get(self, request, *args, **kwargs):
url = file.storage.url(file.name, storage_url=True, http_method=request.method)
protocol = urlparse(url).scheme

# Let NGINX handle it
response = HttpResponse()
# The below header tells NGINX to catch it and serve, see docker-config/nginx-app.conf
redirect = '/file_download/' + protocol + '/' + url.replace(protocol + '://', '')

response['X-Accel-Redirect'] = redirect
response['Content-Disposition'] = 'attachment; filename="{}"'.format(file.name)
return response
# NGINX downloads are a solid way to make uwsgi workers free
if settings.USE_NGINX_FOR_EXPORT_DOWNLOADS:
# let NGINX handle it
response = HttpResponse()
# below header tells NGINX to catch it and serve, see docker-config/nginx-app.conf
redirect = '/file_download/' + protocol + '/' + url.replace(protocol + '://', '')
response['X-Accel-Redirect'] = redirect
response['Content-Disposition'] = 'attachment; filename="{}"'.format(file.name)
response['filename'] = os.path.basename(file.name)
return response

# No NGINX: standard way for export downloads in the community edition
else:
ext = file.name.split('.')[-1]
response = RangedFileResponse(request, file, content_type=f'application/{ext}')
response['Content-Disposition'] = f'attachment; filename="{file.name}"'
response['filename'] = os.path.basename(file.name)
return response
else:
if export_type is None:
file_ = snapshot.file
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Generated by Django 3.2.16 on 2023-04-19 11:36

from django.db import migrations, models


class Migration(migrations.Migration):

dependencies = [
('data_export', '0009_alter_convertedformat_traceback'),
]

operations = [
migrations.AlterField(
model_name='convertedformat',
name='export_type',
field=models.CharField(max_length=64),
),
]
4 changes: 1 addition & 3 deletions label_studio/data_export/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -217,9 +217,7 @@ class Status(models.TextChoices):
help_text='Traceback report in case of errors'
)
export_type = models.CharField(
max_length=64,
choices=Status.choices,
default=Status.CREATED,
max_length=64
)
created_at = models.DateTimeField(
_('created at'),
Expand Down
1 change: 1 addition & 0 deletions label_studio/data_export/serializers.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ class Meta:
expandable_fields = {
'drafts': (AnnotationDraftSerializer, {'many': True}),
'predictions': (PredictionSerializer, {'many': True}),
'annotations': (AnnotationSerializer, {'many': True})
}


Expand Down
108,298 changes: 108,296 additions & 2 deletions label_studio/frontend/dist/react-app/index.js

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion label_studio/frontend/dist/react-app/index.js.map

Large diffs are not rendered by default.

4,029 changes: 4,028 additions & 1 deletion label_studio/frontend/dist/react-app/main.css

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion label_studio/frontend/dist/react-app/main.css.map

Large diffs are not rendered by default.

3 changes: 2 additions & 1 deletion label_studio/frontend/src/assets/icons/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -24,4 +24,5 @@ export { default as IconTerminal } from './terminal.svg';
export { default as LsThumbsDown } from './thumbs-down.svg';
export { default as LsThumbsUp } from './thumbs-up.svg';
export { default as IconUpload } from './upload.svg';
export { default as LsPencil } from './pencil.svg';
export { default as LsPencil } from './pencil.svg';
export { default as IconInfoOutline } from './info-outline.svg';
5 changes: 3 additions & 2 deletions label_studio/frontend/src/components/Card/Card.styl
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
.card
border-radius 5px
background-color rgba(#000, 0.03)
border 1px solid rgba(#000, 0.05)

&__header
display flex
Expand All @@ -9,9 +10,9 @@
align-items center
font-weight 500
font-size 16px
line-height 22px
line-height 18px
justify-content space-between
box-shadow 0 1px 0 0 rgba(#000, 0.15)
box-shadow 0 1px 0 0 rgba(#000, 0.1)

&-content
display flex
Expand Down
2 changes: 1 addition & 1 deletion label_studio/frontend/src/components/Columns/Columns.styl
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,4 @@
font-weight 500
font-size 16px
line-height 22px
padding 0 16px
padding 0 16px 0 0
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
import React from 'react';
import { cn } from '../../utils/bem';
import './DescriptionList.styl';
import { IconInfoOutline } from '../../assets/icons';
import { Tooltip } from '../../components/Tooltip/Tooltip';

export const DescriptionList = ({style, className, children}) => {
return (
Expand All @@ -10,10 +12,16 @@ export const DescriptionList = ({style, className, children}) => {
);
};

DescriptionList.Item = ({ retmClassName, descriptionClassName, term, descriptionStyle, termStyle, children }) => {
DescriptionList.Item = ({ retmClassName, descriptionClassName, term, descriptionStyle, termStyle, children, help }) => {
return (
<>
<dt className={cn('dl').elem('dt').mix(retmClassName)} style={descriptionStyle}>{term}</dt>
<dt className={cn('dl').elem('dt').mix(retmClassName)} style={descriptionStyle}>
{term} {help ? (
<Tooltip style={{ whiteSpace: "pre-wrap" }} title={help}>
<IconInfoOutline className={cn('help-icon')} width="14" height="14" />
</Tooltip>
): "" }
</dt>
<dd className={cn('dl').elem('dd').mix(descriptionClassName)} style={termStyle}>{children}</dd>
</>
);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,17 @@
font-size 16px
line-height 22px
color rgba(#000, 0.6)
grid-template-columns 30% 70%
grid-template-columns 40% 60%
grid-row-gap 12px

&__dt
font-weight 500
min-width 300px

&__dd
margin 0

.help-icon
opacity 0.5
top 1px
position relative
26 changes: 16 additions & 10 deletions label_studio/frontend/src/components/Tooltip/Tooltip.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,12 @@ import { Children, cloneElement, forwardRef, useCallback, useEffect, useMemo, us
import { createPortal } from "react-dom";
import { Block, Elem } from "../../utils/bem";
import { alignElements } from "../../utils/dom";
import { isDefined } from "../../utils/helpers";
import { aroundTransition } from "../../utils/transition";
import "./Tooltip.styl";

export const Tooltip = forwardRef(
({ title, children, defaultVisible, disabled, style }, ref) => {
({ title, children, alignment, defaultVisible, disabled, style }, ref) => {
if (!children || Array.isArray(children)) {
throw new Error("Tooltip does accept a single child only");
}
Expand All @@ -18,18 +19,23 @@ export const Tooltip = forwardRef(
defaultVisible ? "visible" : null,
);
const [injected, setInjected] = useState(false);
const [align, setAlign] = useState('top-center');
const [align, setAlign] = useState(alignment ?? 'top-center');

const calculatePosition = useCallback(() => {
const { left, top, align: resultAlign } = alignElements(
triggerElement.current,
tooltipElement.current,
align,
10,
);
const parent = triggerElement.current;
const target = tooltipElement.current;

setOffset({ left, top });
setAlign(resultAlign);
if (isDefined(parent) && isDefined(target)) {
const { left, top, align: resultAlign } = alignElements(
parent,
target,
align,
10,
);

setOffset({ left, top });
setAlign(resultAlign);
}
}, [triggerElement.current, tooltipElement.current]);

const performAnimation = useCallback(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,9 +43,11 @@ export const StorageCard = ({
setStorageData(storage);
}, [storage]);

const notSyncedYet = synced !== null || ['in_progress', 'queued'].includes(storageData.status);

return (
<Card
header={storageData.title ?? `Untitled ${storageData.type}`}
header={storageData.title.slice(0, 70) ?? `Untitled ${storageData.type}`}
extra={(
<Dropdown.Trigger align="right" content={(
<Menu size="compact" style={{ width: 110 }}>
Expand All @@ -58,21 +60,26 @@ export const StorageCard = ({
)}
>
<StorageSummary
target={target}
storage={storageData}
className={rootClass.elem('summary')}
storageTypes={storageTypes}
/>
<div className={rootClass.elem('sync')}>
<Space size="small">
<Button waiting={syncing} onClick={startSync}>
<div>
<Button
waiting={syncing}
onClick={startSync}
disabled={notSyncedYet}
>
Sync Storage
</Button>
{synced !== null ? (
{notSyncedYet && (
<div className={rootClass.elem('sync-count')}>
Synced {synced} task(s)
Syncing may take some time, please refresh the page to see the current status.
</div>
) : null}
</Space>
)}
</div>
</div>
</Card>
);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ export const StorageSet = ({title, target, rootClass, buttonLabel}) => {
target,
},
}).then(types => {
setStorageTypes(types);
setStorageTypes(types ?? []);
});
}, []);

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import { Block, cn } from '../../../utils/bem';
import { StorageSet } from './StorageSet';
import './StorageSettings.styl';


export const StorageSettings = () => {
const rootClass = cn("storage-settings");

Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
.storage-settings
max-width 680px
padding 40px

&__description
font-size 16px
Expand Down Expand Up @@ -48,5 +49,7 @@
align-items center

&__sync-count
margin-top 14px
line-height 18px
font-size 14px
opacity 0.7
Original file line number Diff line number Diff line change
@@ -1,24 +1,90 @@
import { format } from 'date-fns/esm';
import React from 'react';
import { React } from 'react';
import { DescriptionList } from '../../../components/DescriptionList/DescriptionList';
import { modal } from '../../../components/Modal/Modal';
import { Button } from '../../../components';
import { Oneof } from '../../../components/Oneof/Oneof';
import { lastTwoLines } from '../../../utils/helpers';

export const StorageSummary = ({ target, storage, className, storageTypes = [] }) => {
const storageStatus = storage.status.replace(/_/g, ' ').replace(/(^\w)/, match => match.toUpperCase());

const handleButtonClick = () => {
const msg = `Error logs for ${target==='export' ? 'export ': ''}${storage.type} ` +
`storage ${storage.id} in project ${storage.project} and job ${storage.last_sync_job}:\n\n` +
`${lastTwoLines(storage.traceback)}\n\n` +
`meta = ${JSON.stringify(storage.meta)}\n`;

modal({
title: "Storage error logs",
body: (
<>
<pre style={{ background: "#eee", borderRadius: 5, padding: 10 }}>{msg}</pre>
<Button size="compact" onClick={() => { navigator.clipboard.writeText(msg); }}>Copy</Button>
{(target === 'export' ? (
<a style={{ float: "right" }} target="_blank" href="https://docs.heartex.com/guide/storage.html#Target-storage-permissions">
Check Target Storage documentation
</a>
) : (
<a style={{ float: "right" }} target="_blank" href="https://docs.heartex.com/guide/storage.html#Source-storage-permissions">
Check Source Storage documentation
</a>
)
)}
</>
),
style: { width: "700px" },
optimize: false,
allowClose: true,
});
};

export const StorageSummary = ({storage, className, storageTypes = []}) => {
return (
<div className={className}>
<DescriptionList>
<DescriptionList.Item term="Type">
{storageTypes.find(s => s.name === storage.type)?.title ?? storage.type}
{(storageTypes ?? []).find(s => s.name === storage.type)?.title ?? storage.type}
</DescriptionList.Item>
<Oneof value={storage.type}>
<SummaryS3 case="s3" storage={storage}/>
<SummaryS3 case={["s3", "s3s"]} storage={storage}/>
<GSCStorage case="gcs" storage={storage}/>
<AzureStorage case="azure" storage={storage}/>
<RedisStorage case="redis" storage={storage}/>
<LocalStorage case="localfiles" storage={storage}/>
</Oneof>

<DescriptionList.Item
term="Status"
help={'Initialized: storage was added, but never synced, it is enough to start URI links resolving \n' +
'Queued: sync job is in the queue, but not yet started \n'+
'In progress: sync job is running \n' +
'Failed: sync job stopped, some errors occurred \n' +
'Completed: sync job completed successfully'}
>
{
storageStatus === 'Failed' ? (
<span
style={{ cursor:"pointer", borderBottom: "1px dashed gray" }}
onClick={handleButtonClick}>Failed</span>
) :
storageStatus
}
</DescriptionList.Item>

<DescriptionList.Item
term={target === 'export' ? 'Annotations' : 'Tasks' }
help={
target === 'export' ?
'Number of annotations successfully saved during the last sync':
'Number of new tasks successfully added during the last sync.\n' +
"Tasks that have already been synced won't be added to the project and included in this counter."
}
>
{storage.last_sync_count ? storage.last_sync_count : "0"}
</DescriptionList.Item>

<DescriptionList.Item term="Last Sync">
{storage.last_sync ? format(new Date(storage.last_sync), 'MMMM dd, yyyy ∙ HH:mm:ss') : "Never synced"}
{storage.last_sync ? format(new Date(storage.last_sync), 'MMMM dd, yyyy ∙ HH:mm:ss') : "Not synced yet"}
</DescriptionList.Item>
</DescriptionList>
</div>
Expand Down
Loading