I am looking to fetch SP data using sharepoint Folder based Rest api. When i use filters, it is taking long time (close to one minute) to fetch the data. when i use pages rest api instead of folder level, that is faster. But my need is to hit the folder api. Any help would be appreciated.
Example:
this is faster - https://sharepoint.com/sites/mysite/_api/Web/Lists/ GetByTitle('pages')/items?$ select=field1,field2,field3&$orderby=field5&$filter=field6&$top=10
This is slow - https://sharepoint.com/sites/mysite/_api/web/ GetFolderByServerRelativeUrl('pages/myfolder')/files?$select=ListItemAllFields/field1,ListItemAllFields/field2, ListItemAllFields/field3&$orderby=field5&$filter=ListItemAllFields/field6&$top=10
Do i need to change the way i am querying the folder to improve the performance?
If you are using SharePoint Online, you can use Microsoft Graph API for improved performance.
To get the files in the folder you can use below endpoint.
/me/drive/root:/path/to/folder:/children
I would suggest, you try your queries in Microsoft Graph explorer
You can get further guidance here
Related
I have a large number of documents, word/excel/pdf etc stored across a number of windows file shares, not sure on the total size but will be min a few TBs of files. I need an interface to search these documents (including their contents) and preview/download documents that match the search. It's also important that ACLs are respected, only returning search results for files the logged in user has access to.
The initial idea was to use a tool like Apache Tika to get the file contents/meta data and dump it all into elastic or something similar. The biggest challenge with this idea is respecting the ACLs and filtering search results.
Is there an obvious Office365/Azure solution to this? I'm a newbie with Azure and it's a bit of a minefield but have seen I can use an on premise gateway to connect file share's to power apps and other azure tools. So hoping there's functionality available that will allow me to create a front end to search through these file shares etc.
Two separated questions in here. You can use Azure Search which has indexers capable to extract and index the content of your files with 0 lines of code. However, due the large amount of data, the billing will not be cheap and you'll need several partitions which also increase the cost.
About authentication / authorization, you'll need a frontend to display the results, so you'd better implement the authentication / authorization on it, and let Azure Search only for the query part. You can grant permission just to your front end.
PS: You can use Azure AD for authentication part, but there's no ready to use functionality to assign which information each user can see. You'll need to implement this part
We delivered a successful project a few days back and now we need to make some performance improvements in our WCF Restful API.
The projects is using the following tools/technologies
1- LINQ
2- Entity Framework
3- Enterprise library for Logging/Exception handling
4- MS SQL 2008
5- Deployed on IIS 7
A few things to note
1- 10-20 queries have more than 7 table joins in LINQ
2- The current IIS has more than 10 applications deployed
3- The entity framework has around 60 tables
4- The WCF api is using HTTPS
5- All the API call return JSON responses
The general flow is
1- WCF call is received
2- Session is checked
3- Function from BL layer is called
4- Function from DA layer is called
5- Response returned in JSON
Currently, as per my little knowledge and research I think that the
following might improve performance
1- Implement caching for reference data
2- Move LINQ queries with more than 3 joins to stored procedure (and use hints maybe?)
3- Database table re-indexing
4- Use performance counters to know the problem area's
5- Move functions with more than 3 update/delete/inserts to stored procedure
Can you point out some issue with the above improvements ? and what
other improvements can i do ?
Your post is missing some background on your improvement suggestions. Are they just guesses or have you actually measured and identified them as problem areas?
There really is no substitute for proper performance monitoring and profiling to determine which area you should focus on for optimizations. Everything else is just guesswork, and although some things might be obvious, it's often the not-so-obvious things that actually improve performance.
Run your code through a performance profiling tool to quickly identify problem areas inside the actual application. If you don't have access to the Visual Studio Performance Analyzer (Visual Studio Premium or Ultimate), take a look at PerfView which is a pretty good memory/CPU profiler that won't cost you anything.
Use a tool such as MiniProfiler to be able to easily set up measuring points, as well as monitoring the Entity Framework execution at runtime. MiniProfiler can also be configured to save the results to a database which is handy when you don't have a UI.
Analyzing the generated T-SQL statements from the Entity Framework, which can be seen in MiniProfiler, should allow you to easily measure the query performance by looking at the SQL execution plans as well as fetching the SQL IO statistics. That should give you a good overview of what can/should be put into stored procedures and if you need any other indexes.
We have a need to convert MS Office documents to PDF real time when someone provides a link to a document after checking whether user is authorized to view the document or not for an intranet portal. We also need to cache the documents based on the last modified date of the document, we should not convert the document again if another user requests the same document and the document content is not modified since it was last converted.
I have some basic questions on how we can implement this - and would like to check if anyone has previous experience or thoughts how they see this implemented?
For example, if we choose J2EE as the technology, and choose one of the open source Java libraries for PDF conversion; I have following questions.
If there is a 100 MB document - we would need to download entire document from the system where the document is hosted before we start converting the document. This approach may have major concerns on the response time given that this needs to be real time viewing. Is there an option to read first page of a document without downloading entire document so that we can convert document page by page?
How can we cache a document? I do not think we can either store the document in server or database. The reason is this could lead to anyone who is having access to either database or server - can access document content. Any thoughts?
Or do you suggest any out of the box product to do this instead of custom development?
Thanks
I work for a company that creates a product that does exactly what you are trying to do using Java / .NET Web service calls, so let me see if I can answer your questions without bias.
The whole document will need to be downloaded as it will need to be interpreted before PDF Conversion (e.g. for page numbering purposes) can take place. I am sure you are just giving an example, but 100MB is very large for an MS-Office document, although we do see it from time to time.
You can implement caching based on your exact security requirements. If you don't want to store the converted files in a (secured) DB or file system then perhaps you want to store them on a different server behind a firewall. Depending on the number of documents and size you anticipate you may want to cache them in memory. I am sure there are many J2EE caching libraries available, I know there are plenty in .NET. Just keep the most frequently requested documents in your cache.
Depending on your budget you may go for an out of the box product (hint hint :-). I know there are free libraries available for Java that leverage Open Office, but you get the same formatting limitations when opening MS-Office Files in OO. Be careful when trying to do your own MS-Office integration / automation. It is possible to make it reliable and scalable (we did), but it takes a long time and a lot of work.
I hope this helps.
Good morning.
I am currently working on a project which was originally going to be hosted on a physical server with SQL2k8R2, but it looks like we are moving towards the cloud and Azure... Since SQL Azure does not currently support Full Text Indexing, i have been looking at Lucene.NET with the AzureDirectory project for back end storage. The way this will work is that updates will come in and be queued. once processed, they will be placed in a ToIndex queue, which will kick off Lucene.NET indexing. I am just wondering if there would be a better way of doing this? We dont need to use Azure for this project, so if there is a better solution somewhere, please tell us... main requirement for hosting is it is in Europe...(Azure and Amazon Data centers in Dublin is handy, RackSpace in US is not so handy).
Thanks.
I haven't used that project, but it looks promising. From what I understand, the basic issue is that Lucene requires a file-system. I see 2 other possible solutions (basically just doing what the library does):
Use Azure Drive Storage and a worker role
Use Drive storage, but use a VM (if there are config issues with using a worker role)
http://go.microsoft.com/?linkid=9710117
SQLite also has full text search available, but it has the same basic issue - it requires a filesystem:
http://www.sqlite.org/fts3.html
I have another solution for you, but it's a bit more radical, and a bit more of a conceptual one.
You could create your own indexes, using azure table storage. Create partitions based on each word in your documents, as all tables are indexed on the partitionkey, per word search should be fast, and just do memory joins for multiple word searches.
You could host it as an Azure Website as long as your Lucene index is less than 1GB.
I did this recently when I rewrote Ask Jon Skeet to be hosted as a self contained Azure Website. It uses WebBackgrounder to poll the Stackoverflow API for changes, before updating the Lucene index.
My company is looking into writing a custom application that will need to perform many list item queries across multiple site collections. It will need to run for WSS 3.0 and it 'would be nice' if it worked on WSS 2.0 as well. It won't be designed for MOSS/SPS but again it 'would be nice' if it worked on these platforms. There is no restriction on which .NET version should be used for the solution.
For this type of application, what would be better: the object model/API or SharePoint web services? The primary factor I'm considering is performance, followed by features and functionality. Thanks!
Object model is better as you can gain access to additional features and the full detail of the list items, such as the version history.
The object model is also better for performance (as long as you dispose() your spsite and spweb objects properly).
The Sharepoint object model has some differences between 2 and 3, but if you look at the reference for v2 then it will also work fully with v3.
The web services have not changed at all between v2 and v3, which explains why they do not have any new features of v3.
The reason the object model will win on performance is that you will not be serialising the data as Xml and then transmitting a large chunk of Xml, and then deserialising the Xml. The object model spares your memory and bandwidth.
The first thing to consider is "will my code run on a sharepoint server or remotely ?"
If it's running remotely, you don't
have any choice, use web services
If it's running on a sharepoint
server, I would suggest using object
model, as performance will be
better, you'll have access to more
API and authentication will be
easier (=automatic).
+1 to the other posters.
If you decide to go the OM route then you can compile for both WSS 2.0 and WSS 3.0 from the one source. These should get you started.
Developing for Sharepoint 2003 using Visual Studio 2008?
How to reference two versions of an API?
Can the OM be used inside an Infopath form? Currently I'm using the web services to pull in the list data I want but I would rather use the OM.