In the following simple example 'foo.csv' is a simple 3 column table.
The first column is the line number. The second and third columns are strings.
I create the table using schema auto-detection.
(I've also tried creating a table with a specific schema with similar results.)
I then query it using table.getRows.
The rows returned are not in the order they were inserted.
It took me a while to figure out the order, since it's neither order of insertion nor numerically by the first column.
The sort order is
- Length of second column
- Alphabetically by second column
- Length of third column
- Alphabetically by third column
which is completely useless as far as I'm concerned.
How can I either A) preserve the insertion order or B) sort by the first column?
This seems like a relatively straightforward request but I can't find any documentation on it.
const gcloud = require('google-cloud');
const storage = gcloud.storage();
const bigquery = gcloud.bigquery({ projectId: projectId });
const async = require('async');
const bucket = storage.bucket(storage.bucket);
const file = bucket.file('foo.csv');
const dataset = bigquery.dataset('dataset1');
const table = dataset.table('table1');
async.waterfall([
(callback) => {
table.import(file, {
autodetect: true,
maxBadRecords: 500000,
writeDisposition: 'WRITE_TRUNCATE',
}, callback);
},
(job, apiResponse, callback) => {
async.retry({
times: 20000,
interval: 2000
}, (retryCallback, results) => {
job.getMetadata((err, metadata) => {
let status = metadata.status.state;
retryCallback((status == 'DONE') ? null : status);
});
}, callback);
},
(callback) => {
table.getRows({
autoPaginate: false,
maxResults: 100
}, callback);
},
(rows, nextQuery, info, callback) => {
rows.forEach((row) => {
console.log(JSON.stringify(row));
});
callback(null);
}
], (err) => {
console.log(err);
});
Instead of using table.getRows you can use table.query to pass an SQL query in which you could specify the sort using an Order By clause.
https://googlecloudplatform.github.io/google-cloud-node/#/docs/bigquery/0.3.0/bigquery/table?method=query
Related
There is an array in public users = new BehaviorSubject<User[]>([]).
I want to delete element from this observable and refresh it.
My solution is:
const idRemove = 2;
this.users.next(this.user.getValue().filter((u) => u.id !== idRemove)
But I seem I use wrong way of using RXJS
Toward Idiomatic RxJS
Using subscribe instead of .value.
interface User {
id: number
}
const users$ = new BehaviorSubject<User[]>([
{id:1},
{id:2},
{id:3}
]);
function removeId(idRemove: number) {
users$.pipe(
take(1),
map(us => us.filter(u => u.id !== idRemove))
).subscribe(
users$.next.bind(users$)
);
}
users$.subscribe(us =>
console.log("Current Users: ", us)
);
removeId(2);
removeId(1);
removeId(3);
Output:
Current Users: [ { id: 1 }, { id: 2 }, { id: 3 } ]
Current Users: [ { id: 1 }, { id: 3 } ]
Current Users: [ { id: 3 } ]
Current Users: []
To handle state within RxJS pipes you can use the Scan operator
Useful for encapsulating and managing state. Applies an accumulator (or "reducer function") to each value from the source after an initial state is established -- either via a seed value (second argument), or from the first value from the source.
const { Subject, merge } = rxjs;
const { scan, map } = rxjs.operators;
// This function is used to apply new users to the state of the scan
const usersFn = users => state => users
// This function is used to remove all matching users with the given id from the state of the scan
const removeFn = removeId => state => state.filter(user => user.id !== removeId)
// This Subject represents your old user BehaviorSubject
const users$$ = new Subject()
// This Subject represents the place where this.users.next(this.user.getValue().filter((u) => u.id !== idRemove) was called
const remove$$ = new Subject()
// This is your new user$ Observable that handles a state within its pipe. Use this Observable in all places where you need your user Array instead of the user BehaviorSubject
const user$ = merge(
// When users$$ emits the usersFn is called with the users argument (1. time)
users$$.pipe(map(usersFn)),
// When remove$$ emits the removeFn is called with the removeId argument (1. time)
remove$$.pipe(map(removeFn))
).pipe(
// Either the usersFn or removeFn is called the second time with the state argument (2. time)
scan((state, fn) => fn(state), [])
)
// Debug subscription
user$.subscribe(console.log)
// Test emits
users$$.next([
{id: 1, name: "first"},
{id: 2, name: "second"}
])
remove$$.next(2)
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/7.4.0/rxjs.umd.min.js"></script>
Ben Lesh (main Contributor of RxJS) wrote an anser about why not to use getValue in RxJS: The only way you should be getting values "out of" an Observable/Subject is with subscribe!
Using getValue() is discouraged in general for reasons explained here even though I'm sure there are exception where it's fine to use it. So a better way is subscribing to get the latest value and then changing it:
this.users
.pipe(take(1)) // take(1) will make sure we're not creating an infinite loop
.subscribe(users => {
this.users.next(users.filter((u) => u.id !== idRemove);
});
I have two observables, each one you get the value of a Dto:
this.about.aboutHeInfo().subscribe((heInfo: HemDto) => {
this.uiUtils.openDialogResizable({
hem: heInfo
}, true, AboutComponent).subscribe();
});
this.about.aboutPeInfo().subscribe((peInfo: PeoDto) => {
this.uiUtils.openDialogResizable({
peo: peInfo
}, true, AboutComponent).subscribe();
});
The problem is that when creating both observables two screens are opened because each function creates an apenDialog, how can I merge the two observables and open a single dialog box?
This is the merge I am testing:
const ob1 = this.about.aboutInfo().subscribe((heInfo: HemDto) => {
this.heInfo= back;
});
const ob2 = this.about.aboutQoInfo().subscribe((peInfo: PeoDto) => {
this.peInfo= people;
});
forkJoin([ob1, ob2]).subscribe(() => {
this.uiUtils.openDialogResizable({
back: this.heInfo,
people: this.peInfo
}, true, AboutComponent).subscribe();
});
You shouldn't use subscribe before merge function, merge take observable not what subscribe returns
Example
const ob1 = this.about.aboutInfo()
const ob2 = this.about.aboutQoInfo()
forkJoin([ob1, ob2]).subscribe(([dto1, dto2]) => {
...
});
I am trying to set up react-table but am running into the issue of not being able to call hooks inside regular functions, specifically the useTable hook. Say I have something like this:
const Search = () => {
const [tableRows, changeTableRows] = useState([])
const columns = [
{ Header: 'One', accessor: 'header1'},
{ Header: 'Two', accessor: 'header2'}
]
const { getTableProps, getTableBodyProps, headerGroups, rows, prepareRow } = useTable({ columns, tableRows })
const apiSearch = (field, query) => {
axios.get(`ip/querypath?query=${query}&field=${field}`)
.then(({data}) => {
changeTableRows(data)
})
.catch(err => console.log(err))
return (
/* Render forms to input data and call apiSearch, and render actual table */
)
}
The line with useTable is only being called once, with no initial row data. I need it to update when I make the API call but I can't put a hook inside my apiSearch function. Even if I could, I don't know the right way to do it. This throws a syntax error:
let getTableProps, getTableBodyProps, headerGroups, rows, prepareRow
const apiSearch = (field, query) => {
...
{ getTableProps, getTablyBodyProps, headerGroups, rows, prepareRow } = useTable({ columns, tableRows })
}
I know I'm missing something basic and would appreciate any help.
What can be the best way to resolve the data in GraphQL
Here i have a SeekerType and JobType, JobsType is nested in SeekerType
A Seeker can apply to many Jobs. When Querying for a seeker, One can just query for seeker's data or as well as he can query for nested JobType and can get the jobstype data too.
But the Question is that If One doesn't Query for nested JobType
he won't get the Jobs data but mine Seeker resolver in viewerType would be fetching that data too.
So, while providing data to the seeker query how can i handle that, Either he can only want seeker details or may want the jobs details too.
Shall I use resolver of each nestedType and get the parent object, and fetch the relevant data using fields from parent Object???
The code below is just for illustration and clarification, the question is about the best way to resolve data
ViewerType.js
const Viewer = new GraphQLObjectType({
name: 'Viewer',
fields: () => ({
Seeker: {
type: SeekerConnection,
args: _.assign({
seekerId: { type: GraphQLID },
status: { type: GraphQLString },
shortlisted: { type: GraphQLInt },
}, connectionArgs),
resolve: (obj, args, auth, rootValue) => {
const filterArgs = getFilters(args) || {};
return connectionFromPromisedArray(getSeekers(filterArgs), args)
.then((data) => {
// getSeekers() provides all the data required for SeekerType fields and it's
JobsType fields
data.args = filterArgs;
return data;
}).catch(err => new Error(err));
},
},
}),
});
SeekerType.js
const SeekerType = new GraphQLObjectType({
name: 'SeekerType',
fields: () => ({
id: globalIdField('SeekerType', obj => obj._id),
userId: {
type: GraphQLID,
resolve: obj => obj._id,
},
email: { type: GraphQLString },
password: { type: GraphQLString },
firstName: { type: GraphQLString },
lastName: { type: GraphQLString },
imageLink: { type: GraphQLString },
education: { type: GraphQLString },
address: { type: GraphQLString },
jobs: {
type: new GraphQLList(JobType),
},
}),
interfaces: [nodeInterface],
});
getSeekers() provide complete data as graphql fields format with nested
jobs field data too
const getSeekers = filterArgs => new Promise((resolve, reject) => {
if (Object.keys(filterArgs).length === 0) {
Seeker.find(filterArgs, { password: 0 }, (err, d) => {
if (err) return reject(err);
return resolve(d);
});
} else {
async.parallel([
(callback) => {
filterArgs._id = filterArgs.seekerId;
delete filterArgs.seekerId;
Seeker.find(filterArgs).lean()
.exec((err, d) => {
if (err) return callback(err);
if (err === null && d === null) return callback(null);
callback(null, d);
});
},
(callback) => {
filterArgs.seekerId = filterArgs._id;
delete filterArgs._id;
Applicant.find(filterArgs).populate('jobId').lean()
.exec((err, resp) => {
if (err) return callback(err);
callback(null, resp);
});
},
], (err, data) => {
const cleanedData = {
userData: data[0],
userJobMap: data[1],
};
const result = _.reduce(cleanedData.userData, (p, c) => {
if (c.isSeeker) {
const job = _.filter(cleanedData.userJobMap,
v => _.isEqual(v.seekerId, c._id));
const arr = [];
_.forEach(job, (i) => {
arr.push(i.jobId);
});
const t = _.assign({}, c, { jobs: arr });
p.push(t);
return p;
}
return reject('Not a Seekr');
}, []);
if (err) reject(err);
resolve(result);
// result have both SeekerType data and nested type
JobType data too.
});
}
});
I gather this to be a question about how to prevent overfetching related data...I.e. How not to necessarily request jobs data when querying the seeker.
This might have several motivations for optimization and security.
Considerations:
If the consumer (e.g. Web app) is under your control, you can simply avoid requesting the jobs field when querying seeker. As you may know, this is one of the stated goals of graphql to only return what is needed over the wire to the consumer, to minimize network traffic and do things in one trip. On the backend I would imagine the graphql engine is smart enough not to overfetch jobs data either, if it's not requested.
If your concern is more of security or unintentional overfetching by consumer apps out of your control, for example, then you can solve that by creating seperate queries and limiting access to them even. E.g. One query for seeker and another for seekerWithJobsData.
Another technique to consider is graphql directives which offer an include switch that can be used to conditionally serve specific fields. One advantage of using this technique in your scenario might be to allow a convenient way to conditionally display multiple fields in multiple queries depending on the value of a single boolean e.g. JobSearchFlag=false. Read here for an overview of directives: http://graphql.org/learn/queries/
I am not sure I completely understand the question but it seems to me you're loading both seeker and job types info on one level. You should load both of them on demand.
On the seeker level, you only get the seeker information, and you can get the IDs of any records related to that seeker. For example, job types ids (if a seeker has many job types)
On the job type level, when used as a nested level for one seeker, you can use those ids to fetch the actual records. This would make the fetching of job types record on-demand when the query asks for it.
The ID to record fetching can be cached and batched with a library like dataloader
I am new to cyclejs and rxjs in general and was hoping someone could help me solve my problem.
I am trying to build a demo application for my understanding and stuck with rendering JSON objects on the DOM.
My demo application calls the NASA near earth objects API for the past 7 days and tries to display them.
There is a Load More button at the bottom which on clicking will load data of the previous 7 days (Today - 7 upto Today - 14).
The response I get from the API is as follows
{
"links" : {
"next" : "https://api.nasa.gov/neo/rest/v1/feed?start_date=2016-09-06&end_date=2016-09-12&detailed=false&api_key=DEMO_KEY",
"prev" : "https://api.nasa.gov/neo/rest/v1/feed?start_date=2016-08-25&end_date=2016-08-31&detailed=false&api_key=DEMO_KEY",
"self" : "https://api.nasa.gov/neo/rest/v1/feed?start_date=2016-08-31&end_date=2016-09-06&detailed=false&api_key=DEMO_KEY"
},
"element_count" : 39,
"near_earth_objects" : {
"2016-09-06" : [{
some data
},
{
some data
}],
2016-08-31: [{...}],
...
}
}
I am interested in near_earth_objects JSON object but I am unable to map it beacause of it being an Object.
How do I handle such a situations? Below is the code that I have
function main(sources) {
const api_key = "DEMO_KEY";
const clickEvent$ = sources.DOM.select('.load-more').events('click');
const request$ = clickEvent$.map(() => {
return {
url: "https://api.nasa.gov/neo/rest/v1/feed?start_date=2015-09-06&end_date=2016-09-13&api_key=" + api_key,
method: "GET"
}
}).startWith({
url: "https://api.nasa.gov/neo/rest/v1/feed?start_date=2016-08-31&end_date=2016-09-06&api_key=" + api_key,
method: "GET"
});
const response$$ = sources.HTTP.filter(x$ => x$.url.indexOf("https://api.nasa.gov/neo/rest/v1/feed") != -1).select(response$$);
const response$ = response$$.switch(); //flatten the stream
const nasa$ = response$.map(response => {
return response.body
});
const sinks = {
DOM: nasa$.map(nasa =>
([nasa.near_earth_objects]).map(objects => {
var vdom = [];
//I am not very happy with this part. Can this be improved?
for (var key in objects) {
if (objects.hasOwnProperty(key)) {
vdom.push(objects[key].map(obj => div([
h1(obj.name)
])))
}
}
//returning the vdom does not render on the browser. vdom is an array of arrays. How should i correct this?
console.log(vdom);
return vdom;
})
),
HTTP: request$
};
return sinks;
};
Conceptually, you want to extract the entries of nasa.near_earth_objects (i.e., turn the Object into an Array), then flat map that Array into an Observable sequence.
I'll assume you're already using lodash in your project (you can do it without lodash, but you'll just need to write more glue code manually). I'll also assume you're importing RxJS' Observable as Rx.Observable; adjust the names below to suite your code.
You can accomplish the first task using _.toPairs(nasa.near_earth_objects), and the second part by calling .flatMap(), and returning Rx.Observable.from(near_objects). The resulting Observable will emit items for each key in nasa.near_earth_objects. Each item will be an array, with item[0] being the item's key (e.g., 2016-09-06) and item[1] being the item's value.
Using that idea, you can replace your DOM sink with something like:
nasa$.map(nasa => _.toPairs(nasa.near_earth_objects))
.flatMap(near_objects => Rx.Observable.from(near_objects))
.map(near_object => div([
h1(near_object[1].name)
]))
),