How to delete all attributes from the schema in solr? - windows

Deleting all documents from solr is
curl http://localhost:8983/solr/trans/update?commit=true -d "<delete><query>*:*</query></delete>"
Adding a (static) attribute to the schema is
curl -X POST -H 'Content-type:application/json' --data-binary '{ "add-field":{"name":"trans","type":"string","stored":true, "indexed":true},}' http://localhost:8983/solr/trans/schema
Deleting one attribute is
curl -X POST -H 'Content-type:application/json' -d '{ "delete-field":{"name":"trans"}}' http://arteika:8983/solr/trans/schema
Is there a way to delete all attributes from the schema?

At least in version 6.6 of the Schema API and up to the current version 7.5 of it, you can pass multiple commands in a single post (see 6.6 and 7.5 documenation, respectively). There are multiple accepted formats, but the most intuitive one (I think) is just passing an array for the action you want to perform:
curl -X POST -H 'Content-type: application/json' -d '{
"delete-field": [
{"name": "trans"},
{"name": "other_field"}
]
}' 'http://arteika:8983/solr/trans/schema'
So. How do we obtain the names of the fields we want to delete? That can be done by querying the Schema:
curl -X GET -H 'Content-type: application/json' 'http://arteika:8983/solr/trans/schema'
In particular, the copyFields, dynamicFields and fields keys in the schema object in the response.
I automated clearing all copy field rules, dynamic field rules and fields as follows. You can of course use any kind of script that is available to you. I used Python 3 (might work with Python 2, I did not test that).
import json
import requests
# load schema information
api = 'http://arteika:8983/solr/trans/schema'
r = requests.get(api)
# delete copy field rules
names = [(o['source'], o['dest']) for o in r.json()['schema']['copyFields']]
payload = {'delete-copy-field': [{'source': name[0], 'dest': name[1]} for name in names]}
requests.post(api, data = json.dumps(payload),
headers = {'Content-type': 'application/json'})
# delete dynamic fields
names = [o['name'] for o in r.json()['schema']['dynamicFields']]
payload = {'delete-dynamic-field': [{'name': name} for name in names]}
requests.post(api, data = json.dumps(payload),
headers = {'Content-type': 'application/json'})
# delete fields
names = [o['name'] for o in r.json()['schema']['fields']]
payload = {'delete-field': [{'name': name} for name in names]}
requests.post(api, data = json.dumps(payload),
headers = {'Content-type': 'application/json'})
Just a note: I received status 400 responses at first, with null error messages. Had a bit of a hard time figuring out how to fix those, so I'm sharing what worked for me. Changing the default of updateRequestProcessorChain in solrconfig.xml to false (default="${update.autoCreateFields:false}") and restarting the Solr service made those errors go away for me. The fields I was deleting were created automatically, that may have something to do with that.

Related

Curl/GraphQL command failing with 200

I am trying to write a shell script that executes a curl against a GraphQL API and I've never interacted with GQL before. I am getting some strange errors and although I understand this community doesn't have access to the GQL server I was hoping someone could take a look at the script and make sure I'm not doing anything flagrantly wrong syntax-wise (both in the shell script layer as well as the GQL query itself).
My script:
#!/bin/bash
BSEE_WEB_SERVER_DNS=https://mybsee.example.com
BSEE_API_KEY=abc123
siteId=1
scanConfigId=456
runScanQuery='mutation CreateScheduleItem { create_schedule_item(input: {site_id: "$siteId" scan_configuration_ids: "$scanConfigId"}) { schedule_item { id } } }'
runScanVariables='{ "input": "site_id": $scanId }}'
runScanOperationName='CreateScheduleItem'
curl -i --request POST \
--url $BSEE_WEB_SERVER_DNS/graphql/v1 \
--header "Authorization: $BSEE_API_KEY" \
--header 'Content-Type: application/json' \
--data '{"query":"$runScanQuery","variables":{$runScanVariables},"operationName":"${runScanOperationName}"}'
And the output when I run the script off the terminal:
HTTP/2 200
<OMITTED RESPONSE HEADERS>
{"errors":[{"message":"Invalid JSON : Unexpected character (\u0027$\u0027 (code 36)): was expecting double-quote to start field name, Line 1 Col 38","extensions":{"code":3}}]}%
I am omitting the HTTP response headers for security and brevity reasons.
I am wondering if my use of quotes/double-quotes is somehow wrong, or if there is anything about the nature of the GQL query itself (via curl) that looks off to anyone.
I verified with the team that manages the server that the HTTP 200 OK response code is correct. 200 shows that the request succeeded to the GQL API, but that GQL is responding with this error to indicate the query itself is incorrect.
We need to modify the GraphQL bits and fix the bash string quoting.
runScanQuery GraphQL operation
Fix the GraphQL syntax. Use a GraphQL operation name CreateScheduleItem with variables $site_id in the arguments input: { site_id: $siteId, scan_configuration_ids: $scanConfigId:
mutation CreateScheduleItem($site_id: String!, $scanConfigId: String!) {
create_schedule_item(
input: { site_id: $siteId, scan_configuration_ids: $scanConfigId }
) {
schedule_item {
id
}
}
}
runScanVariables: JSON
Our mutation expects two variables, which GraphQL will substitute into CreateScheduleItem($site_id: String!, $scanConfigId: String!). Provide the GraphQL variables as JSON. Here is the expected output after bash variable substitution:
{ "$site_id": "1", "$scanConfigId": "456" }
Get the bash quoting right
Finally, translate the inputs into bash-friendly syntax:
runScanQuery='mutation CreateScheduleItem($site_id: String!, $scanConfigId: String!) { create_schedule_item(input: {site_id: $siteId scan_configuration_ids: $scanConfigId}) { schedule_item { id } } }'
runScanVariables='{"$site_id":"'"$siteId"'","$scanConfigId":"'"$scanConfigId"'"}' # no spaces!
runScanOperationName='CreateScheduleItem'
data='{"query":"'"$runScanQuery"'","variables":'$runScanVariables',"operationName":"'"$runScanOperationName"'"}'
Check our bash formats. Paste the terminal output into a code-aware editor like VSCode. Expect the editor to parse the output correctly.
echo $runScanQuery # want string in graphql format
echo $runScanVariables # want JSON
echo $data # want JSON
Edit: add a public API example
Here's a complete working example using the public Star Wars API:
#!/bin/bash
filmId=1
data='{"query":"query Query($filmId: ID) { film(filmID: $filmId) { title }}","variables":{"filmId":"'"$filmId"'"}}'
curl --location --request POST 'https://swapi-graphql.netlify.app/.netlify/functions/index' \
--header 'Content-Type: application/json' \
--data "$data"
Responds with {"data":{"film":{"title":"A New Hope"}}}.
In GraphQL it's normal to always have 200 status code; client must check response body searching for failures.
The reason is simple: In REST, http is part of the protocol and status code has semantics but in GraphQL http is not part of the protocol, you can have GraphQL over serveral transport protocols:
http: typical scenario docs
WebSocket: does not provide any "status code like" payload. sample
MQTT: does not provide any "status code like" payload
...
The only way that server tells you something (even failures) is the body.
In your case I suggest you jq to parse json via bash script searching error property.
Your error is completely unrelated to GraphQL. You really have wrong JSON.
Error message says Unexpected character (\u0027$\u0027 (code 36)): was expecting double-quote to start field name, Line 1 Col 38",
You can replace escaped \u0027 with apostrophe and you will get
Unexpected character ('$' (code 36)): was expecting double-quote to start field name, Line 1 Col 38",
So it hates dollar sign at position 38 in what you send as data to curl
data='{"query":"'"$runScanQuery"'","variables":'$runScanVariables'
^
this
First - all field names and values in JSON should be wrapped with double quotes, not single.
Second - if you want curl to expand env variable, put it to double quotes, not single.

API to get price of Binance Smart Chain token on PancakeSwap

I have address of a token and I need to get its price in BUSD or BNB.
It's not a problem to use paid API, if there is no other way. This token may not be listed on popular listings so it would be nice to get price somehow directly from PancakeSwap.
Here is a way to get it directly from PancakeSwap
https://api.pancakeswap.info/api/v2/tokens/0x8076c74c5e3f5852037f31ff0093eeb8c8add8d3
A friend of mine used Moralis.
https://docs.moralis.io/introduction/readme
https://docs.moralis.io/moralis-dapp/web3-api/token#gettokenprice
Maybe you can already do something with the documentation, I have otherwise asked my colleague for example code
curl -X 'GET' \
'https://deep-index.moralis.io/api/v2/erc20/0x42F6f551ae042cBe50C739158b4f0CAC0Edb9096/price?chain=bsc&exchange=PancakeSwapv2' \
-H 'accept: application/json' \
-H 'X-API-Key: MY-API-KEY'
Result:
{
"nativePrice": {
"value": "8409770570506626",
"decimals": 18,
"name": "Ether",
"symbol": "ETH"
},
"usdPrice": 19.722370676,
"exchangeAddress": "0x1f98431c8ad98523631ae4a59f267346ea31f984",
"exchangeName": "Uniswap v3"
}
Greetings.
Alternatively, if you are using React you can try the following package: react-pancakeswap-token-price
you can scrape charts.bogged.finance or poocoin.app

Backup and restore some records of an elasticsearch index

I wish to take a backup of some records(eg latest 1 million records only) of an Elasticsearch index and restore this backup on a different machine. It would be better if this could be done using available/built-in Elasticsearch features.
I've tried Elasticsearch snapshot and restore (following code), but looks like it takes a backup of the whole index, and not selective records.
curl -H 'Content-Type: application/json' -X PUT "localhost:9200/_snapshot/es_data_dump?pretty=true" -d '
{
"type": "fs",
"settings": {
"compress" : true,
"location": "es_data_dump"
}
}'
curl -H 'Content-Type: application/json' -X PUT "localhost:9200/_snapshot/es_data_dump/snapshot1?wait_for_completion=true&pretty=true" -d '
{
"indices" : "index_name",
"type": "fs",
"settings": {
"compress" : true,
"location": "es_data_dump"
}
}'
The format of backup could be anything, as long as it can be successfully restored on a different machine.
you can use _reinex API. it can take any query. after reindex, you have a new index as backup, which contains requested records. easily copy it where ever you want.
complete information is here: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html
In the end, I fetched the required data using python driver because that is what I found the easiest for the given use case.
For that, I ran an Elasticsearch query and stored its response in a file in newline-separated format and then I later restored data from it using another python script. A maximum of 10000 entries are returned this way along with the scroll ID to be used to fetch next 10000 entries and so on.
es = Elasticsearch(timeout=30, max_retries=10, retry_on_timeout=True)
page = es.search(index=['ct_analytics'], body={'size': 10000, 'query': _query, 'stored_fields': '*'}, scroll='5m')
while len(page['hits']['hits']) > 0:
es_data = page['hits']['hits'] #Store this as you like
scrollId = page['_scroll_id']
page = es.scroll(scroll_id=scrollId, scroll='5m')

Create inventory Square API / UNIREST Picture Upload

I want to create a product on my website and have it be created on square (which is working). However I also want to set the initial inventory which is seems there is no way to do it from the documentation. https://docs.connect.squareup.com/api/connect/v1/#post-inventory-variationid
If I go into my square account I can manually set up an initial amount, then query that entry and get the id and update it, but who wants to do anything manually. It defeats the purpose. Is there a way to create an inventory entry?
My second struggle is with uploading an image using unirest.
function uploadItemImage($itemId, $image_file)
{
global $accessToken, $locationId, $connectHost;
$requestHeaders = array
(
'Authorization' => 'Bearer ' . $accessToken,
'Accept' => 'application/json',
'Content-Type' => 'multipart/form-data;'
);
$request_body = array
(
'image_data'=>Unirest\Request\Body::file($image_file, 'text/plain', myproduct.jpg')
);
$response = Unirest\Request::post($connectHost . '/v1/' . $locationId . '/items/'.$itemId.'/image', $requestHeaders, $request_body);
print(json_encode($response->type, JSON_PRETTY_PRINT));
}
where $itemId is taken from the product created earlier and $image_file is the direct link to the file on my server
I keep getting this error...
> PHP Fatal error: Uncaught exception 'Unirest\Exception' with message
> 'couldn't open file "https://somewebsite/myPicture.jpg" ' in
> rootFolder/Unirest/Request.php:479 Stack trace:
> #0 rootFolder/Unirest/Request.php(292): Unirest\Request::send('POST', 'https://connect...', Array, Array, NULL, NULL)
> #1 rootFolder/
Any help is much appreciated!
Way to maximise the use of your question!
There is not currently a way to set initial inventory via API, but new item and inventory management APIs are in the works, read more on the Square Blog
I'm assuming that you are not literally using "https://somewebsite/myPicture.jpg" but it seems like unirest thinks you are trying to use a web url instead of getting a file from your filesystem. Try the following curl command and see if you can match up all the parts to unirest:
:)
curl --request POST \
--url https://connect.squareup.com/v1/XXXXXX/items/XXXXX/image \
--header 'authorization: Bearer sq0atp-XXXXX' \
--header 'cache-control: no-cache' \
--header 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \
--form image_data=#/Users/ManuEng13/Desktop/test.png

Couchdb view Queries

Could you please help me in creating a view. Below is the requirement
select * from personaccount where name="srini" and user="pup" order by lastloggedin
I have to send name and user as input to the view and the data should be sorted by lastloggedin.
Below is the view I have created but it is not working
{
"language": "javascript",
"views": {
"sortdatetimefunc": {
"map": "function(doc) {
emit({
lastloggedin: doc.lastloggedin,
name: doc.name,
user: doc.user
},doc);
}"
}
}
}
And this the curl command iam using:
http://uta:password#localhost:5984/personaccount/_design/checkdatesorting/_view/sortdatetimefunc?key={\"name:srini\",\"user:pup\"}
My Questions are
As sorting will be done on key and I want it on lastloggedin so I have given that also in emit function.
But iam passing name and user only as parameters. Do we need to pass all the parameters which we give it in key?
First of all I want to convey to you for the reply, I have done the same and iam getting errors. Please help
Could you please try this on your PC, iam posting all the commands :
curl -X PUT http://uta:password#localhost:5984/person-data
curl -X PUT http://uta:password#localhost:5984/person-data/srini -d '{"Name":"SRINI", "Idnum":"383896", "Format":"NTSC", "Studio":"Disney", "Year":"2009", "Rating":"PG", "lastTimeOfCall": "2012-02-08T19:44:37+0100"}'
curl -X PUT http://uta:password#localhost:5984/person-data/raju -d '{"Name":"RAJU", "Idnum":"456787", "Format":"FAT", "Studio":"VFX", "Year":"2010", "Rating":"PG", "lastTimeOfCall": "2012-02-08T19:50:37+0100"}'
curl -X PUT http://uta:password#localhost:5984/person-data/vihar -d '{"Name":"BALA", "Idnum":"567876", "Format":"FAT32", "Studio":"YELL", "Year":"2011", "Rating":"PG", "lastTimeOfCall": "2012-02-08T19:55:37+0100"}'
Here's the view as you said I created :
{
"_id": "_design/persondestwo",
"_rev": "1-0d3b4857b8e6c9e47cc9af771c433571",
"language": "javascript",
"views": {
"personviewtwo": {
"map": "function (doc) {\u000a emit([ doc.Name, doc.Idnum, doc.lastTimeOfCall ], null);\u000a}"
}
}
}
I have fired this command from curl command :
curl -X GET http://uta:password#localhost:5984/person-data/_design/persondestwo/_view/personviewtwo?startkey=["SRINI","383896"]&endkey=["SRINI","383896",{}]descending=true&include_docs=true
I got this error :
[4] 3000
curl: (3) [globbing] error: bad range specification after pos 99
[5] 1776
[6] 2736
[3] Done descending=true
[4] Done(3) curl -X GET http://uta:password#localhost:5984/person-data/_design/persondestwo/_view/personviewtwo?startkey=["SRINI","383896"]
[5] Done endkey=["SRINI","383896"]
I am not knowing what this error is.
I have also tried passing the parameters the below way and it is not helping
curl -X GET http://uta:password#localhost:5984/person-data/_design/persondestwo/_view/personviewtwo?key={\"Name\":\"SRINI\",\"Idnum\": \"383896\"}&descending=true
But I get different errors on escape sequences
Overall I just want this query to be satisfied through the view :
select * from person-data where Name="SRINI" and Idnum="383896" orderby lastTimeOfCall
My concern is how to pass the multiple parameters from curl command as I get lot of errors if I do the above way.
First off, you need to use an array as your key. I would use:
function (doc) {
emit([ doc.name, doc.user, doc.lastLoggedIn ], null);
}
This basically outputs all the documents in order by name, then user, then lastLoggedIn. You can use the following URL to query.
/_design/checkdatesorting/_view/sortdatetimefunc?startkey=["srini","pup"]&endkey=["srini","pup",{}]&include_docs=true
Second, notice I did not output doc as the value of your query. It takes up much more disk space, especially if your documents are fairly large. Just use include_docs=true.
Lastly, refer to the CouchDB Wiki, it's pretty helpful.
I just stumbled upon this question. The errors you are getting are caused by not escaping this command:
curl -X GET http://uta:password#localhost:5984/person-data/_design/persondestwo/_view/personviewtwo?startkey=["SRINI","383896"]&endkey=["SRINI","383896",{}]descending=true&include_docs=true
The & character has a special meaning on the command-line and should be escaped when part of an actual parameter.
So you should put quotes around the big URL, and escape the quotes inside it:
curl -X GET "http://uta:password#localhost:5984/person-data/_design/persondestwo/_view/personviewtwo?startkey=[\"SRINI\",\"383896\"]&endkey=[\"SRINI\",\"383896\",{}]descending=true&include_docs=true"

Resources