I'm a newbee Ceph developer, and recenlty reading code of snapshots. From
pg_pool_t::add_unmanaged_snap, it's obvious that the first RBD snapshot id should
start from 2, but in reality, it starts from 4, I wonder whether there are some organisms
in RBD snap, which increments snap_seq, could anyone help me?
Thanks in advance!
Below is
the code of pg_pool_t::add_unmanaged_snap.
void pg_pool_t::add_unmanaged_snap(uint64_t& snapid)
{
ceph_assert(!is_pool_snaps_mode());
if (snap_seq == 0) {
// kludge for pre-mimic tracking of pool vs selfmanaged snaps. after
// mimic this field is not decoded but our flag is set; pre-mimic, we
// have a non-empty removed_snaps to signifiy a non-pool-snaps pool.
removed_snaps.insert(snapid_t(1));
snap_seq = 1;
}
flags |= FLAG_SELFMANAGED_SNAPS;
snapid = snap_seq = snap_seq + 1;
}
Following screenshot is the process of rbd snapshot creation on a totally new rbd pool. Obviously, snapshot id here starts from 4
rbd snapshot creation on a totally new rbd pool
Related
EDIT:
I have heavily edited this question after making some significant new discoveries and the question not having any answers yet.
Historically/AFAIK, keeping your Mac awake while in closed-display mode and not meeting Apple's requirements, has only been possible with a kernel extension (kext), or a command run as root. Recently however, I have discovered that there must be another way. I could really use some help figuring out how to get this working for use in a (100% free, no IAP) sandboxed Mac App Store (MAS) compatible app.
I have confirmed that some other MAS apps are able to do this, and it looks like they might be writing YES to a key named clamshellSleepDisabled. Or perhaps there's some other trickery involved that causes the key value to be set to YES? I found the function in IOPMrootDomain.cpp:
void IOPMrootDomain::setDisableClamShellSleep( bool val )
{
if (gIOPMWorkLoop->inGate() == false) {
gIOPMWorkLoop->runAction(
OSMemberFunctionCast(IOWorkLoop::Action, this, &IOPMrootDomain::setDisableClamShellSleep),
(OSObject *)this,
(void *)val);
return;
}
else {
DLOG("setDisableClamShellSleep(%x)\n", (uint32_t) val);
if ( clamshellSleepDisabled != val )
{
clamshellSleepDisabled = val;
// If clamshellSleepDisabled is reset to 0, reevaluate if
// system need to go to sleep due to clamshell state
if ( !clamshellSleepDisabled && clamshellClosed)
handlePowerNotification(kLocalEvalClamshellCommand);
}
}
}
I'd like to give this a try and see if that's all it takes, but I don't really have any idea about how to go about calling this function. It's certainly not a part of the IOPMrootDomain documentation, and I can't seem to find any helpful example code for functions that are in the IOPMrootDomain documentation, such as setAggressiveness or setPMAssertionLevel. Here's some evidence of what's going on behind the scenes according to Console:
I've had a tiny bit of experience working with IOMProotDomain via adapting some of ControlPlane's source for another project, but I'm at a loss for how to get started on this. Any help would be greatly appreciated. Thank you!
EDIT:
With #pmdj's contribution/answer, this has been solved!
Full example project:
https://github.com/x74353/CDMManager
This ended up being surprisingly simple/straightforward:
1. Import header:
#import <IOKit/pwr_mgt/IOPMLib.h>
2. Add this function in your implementation file:
IOReturn RootDomain_SetDisableClamShellSleep (io_connect_t root_domain_connection, bool disable)
{
uint32_t num_outputs = 0;
uint32_t input_count = 1;
uint64_t input[input_count];
input[0] = (uint64_t) { disable ? 1 : 0 };
return IOConnectCallScalarMethod(root_domain_connection, kPMSetClamshellSleepState, input, input_count, NULL, &num_outputs);
}
3. Use the following to call the above function from somewhere else in your implementation:
io_connect_t connection = IO_OBJECT_NULL;
io_service_t pmRootDomain = IOServiceGetMatchingService(kIOMasterPortDefault, IOServiceMatching("IOPMrootDomain"));
IOServiceOpen (pmRootDomain, current_task(), 0, &connection);
// 'enable' is a bool you should assign a YES or NO value to prior to making this call
RootDomain_SetDisableClamShellSleep(connection, enable);
IOServiceClose(connection);
I have no personal experience with the PM root domain, but I do have extensive experience with IOKit, so here goes:
You want IOPMrootDomain::setDisableClamShellSleep() to be called.
A code search for sites calling setDisableClamShellSleep() quickly reveals a location in RootDomainUserClient::externalMethod(), in the file iokit/Kernel/RootDomainUserClient.cpp. This is certainly promising, as externalMethod() is what gets called in response to user space programs calling the IOConnectCall*() family of functions.
Let's dig in:
IOReturn RootDomainUserClient::externalMethod(
uint32_t selector,
IOExternalMethodArguments * arguments,
IOExternalMethodDispatch * dispatch __unused,
OSObject * target __unused,
void * reference __unused )
{
IOReturn ret = kIOReturnBadArgument;
switch (selector)
{
…
…
…
case kPMSetClamshellSleepState:
fOwner->setDisableClamShellSleep(arguments->scalarInput[0] ? true : false);
ret = kIOReturnSuccess;
break;
…
So, to invoke setDisableClamShellSleep() you'll need to:
Open a user client connection to IOPMrootDomain. This looks straightforward, because:
Upon inspection, IOPMrootDomain has an IOUserClientClass property of RootDomainUserClient, so IOServiceOpen() from user space will by default create an RootDomainUserClient instance.
IOPMrootDomain does not override the newUserClient member function, so there are no access controls there.
RootDomainUserClient::initWithTask() does not appear to place any restrictions (e.g. root user, code signing) on the connecting user space process.
So it should simply be a case of running this code in your program:
io_connect_t connection = IO_OBJECT_NULL;
IOReturn ret = IOServiceOpen(
root_domain_service,
current_task(),
0, // user client type, ignored
&connection);
Call the appropriate external method.
From the code excerpt earlier on, we know that the selector must be kPMSetClamshellSleepState.
arguments->scalarInput[0] being zero will call setDisableClamShellSleep(false), while a nonzero value will call setDisableClamShellSleep(true).
This amounts to:
IOReturn RootDomain_SetDisableClamShellSleep(io_connect_t root_domain_connection, bool disable)
{
uint32_t num_outputs = 0;
uint64_t inputs[] = { disable ? 1 : 0 };
return IOConnectCallScalarMethod(
root_domain_connection, kPMSetClamshellSleepState,
&inputs, 1, // 1 = length of array 'inputs'
NULL, &num_outputs);
}
When you're done with your io_connect_t handle, don't forget to IOServiceClose() it.
This should let you toggle clamshell sleep on or off. Note that there does not appear to be any provision for automatically resetting the value to its original state, so if your program crashes or exits without cleaning up after itself, whatever state was last set will remain. This might not be great from a user experience perspective, so perhaps try to defend against it somehow, for example in a crash handler.
I am trying to make asynchronous kernel calls to my GPGPU using CUDAfy .NET.
When I pass values to the kernel and copy them back to the host, I do not always get the value I expect.
I have a structure Foo with a byte Bar:
[Cudafy]
public struct Foo {
public byte Bar;
}
And I have a kernel I want to call:
[Cudafy]
public static void simulation(GThread thread, Foo[] f)
{
f[0].Bar = 3;
thread.SyncThreads();
}
I have a single thread with streamID = 1 (I tried using multiple threads, and noticed the issue. Reducing to a single thread didn't seem to fix the issue though).
//allocate
streamID = 1;
count = 1;
gpu.CreateStream(streamID);
Foo[] sF = new Foo[count];
IntPtr hF = gpu.HostAllocate<Foo>(count);
Foo[] dF = gpu.Allocate<Foo>(sF);
while (true)
{
//set value
sF[0].Bar = 1;
byte begin = sF[0].Bar;
//host -> pinned
GPGPU.CopyOnHost<Foo>(sF, 0, hF, 0, count);
sF[0].Bar = 2;
lock (gpu)
{
//pinned -> device
gpu.CopyToDeviceAsync<Foo>(hF, 0, dF, 0, count, streamID);
//run
gpu.Launch().simulation(dF);
//device -> pinned
gpu.CopyFromDeviceAsync<Foo>(dF, 0, hF, 0, count, streamID);
}
//WAIT
gpu.SynchronizeStream(streamID);
//pinned -> host
GPGPU.CopyOnHost<Foo>(hF, 0, sF, 0, count);
byte end = sF[0].Bar;
}
//de-allocate
gpu.Free(dF);
gpu.HostFree(hF);
gpu.DestroyStream(streamID);
First I create a stream on the GPU.
I am creating a regular structure Foo array of size 1 (sF) and setting it's Bar value to 1. Then I create pinned memory on the host (hF) for Foo as well. I also create memory on the device for Foo (dF).
I initialize the structure's Bar value to 1 then I copy it to the pinned memory (As a check, I set the value to 2 for the structure after copying to pinned, you'll see why later). Then I use a lock to ensure I have full access to the GPU and I queue a copy to dF, a run for the kernel, and a copy from dF. At this point I don't know when this will all actually run on the GPU... so I can call SynchronizeStream to wait on the host until the device is done.
When it's done, I can copy the pinned memory (hF) to the shared memory (sF). When I get the value, it's usually a 3 (which was set on the device) or a 1 (which means either the value wasn't set in the kernel, or the new value wasn't copied to the pinned memory). I do know that the pinned memory is copied to the structure because the structure never has the value of 2.
Over many runs, a small percentage is runs results in something other than begin=1 and end=3. It would always be begin=1, end=1 and it happens about 5-10% of the time.
I have no idea why this happens. I know it generally highlights a race condition, but by calling the sync calls, I would expect the async calls to work in a predictable fashion.
Why would I be encountering this kind of issue with this code?
Thank you so much!
-Phil
I just figured out the issue that was occurring. While the launch was being done asynchronously... I didn't include the stream for the launch.
Changing my launch to be:
gpu.Launch(gridsize,blocksize,streamID).simulation(dF);
resolved the problem. It seems that the launches were occurring on stream 0 and the stream 1 and 2 were being synced. So sometimes the data gets set, sometimes it doesn't. A race condition.
I'm trying to integrate Realm into my project and noticed an issue. I've seen other posts on this, but they were a little over a year ago and have been resolved..
When adding objects to Realm, things are file. But when removing objects, they get removed from the DB, but the file size is still large. If I open the realm file in TextEdit, I can see raw text of old records. Why aren't they getting fully deleted?
Take a look at this screenshot. Zero files in the Realm DB, but the file size is 23 mb.
Thanks.
as bcamur sad,
the Realm file will maintain its size on disk to efficiently reuse
that space for future objects
but there is also written
The extra space will eventually be reused by future writes, or may be
compacted — for example by calling
Realm().writeCopyToPath(_:encryptionKey:).
and
call invalidate to tell Realm that you no longer need any of the
objects that you’ve read from the Realm so far, which frees us from
tracking intermediate versions of those objects. The Realm will update
to the latest version the next time it is accessed
I've also realised that my Realm file size grows too big (and it never decreased) and the solution for me was to initialise my Realm db in the following way:
class RealmManager {
static let shared = RealmManager()
private var realm: Realm?
private init() {
let config = Realm.Configuration(schemaVersion: 1, shouldCompactOnLaunch: { totalBytes, usedBytes in
// totalBytes refers to the size of the file on disk in bytes (data + free space)
// usedBytes refers to the number of bytes used by data in the file
// Compact if the file is over 100MB in size and less than 50% 'used'
let oneHundredMB = 100 * 1024 * 1024
return (totalBytes > oneHundredMB) && (Double(usedBytes) / Double(totalBytes)) < 0.5
})
do {
// Realm is compacted on the first open if the configuration block conditions were met.
realm = try Realm(configuration: config)
} catch let error {
// handle error compacting or opening Realm
print(error)
}
}
}
The key is to add shouldCompactOnLaunch block to my Configuration and please, note that the compaction operation will not be completed till you have another process accessing your realm database (e.g.: an opened db in Realm Studio).
For more information, you can check the following link:
https://realm.io/docs/swift/latest/#compacting-realms
Realm holds on to that space to use later on for new objects:
You can also delete all objects stored in a Realm. Note the Realm file will maintain its size on disk to efficiently reuse that space for future objects.
See this part of documentation
Swift Version 3.0.1
For compact your DB:
func compactRealm() {
let defaultURL = Realm.Configuration.defaultConfiguration.fileURL!
let defaultParentURL = defaultURL.deletingLastPathComponent()
let compactedURL = defaultParentURL.appendingPathComponent("default-compact.realm")
autoreleasepool {
let realm = try! Realm()
try! realm.writeCopy(toFile: compactedURL)
try! FileManager.default.removeItem(at: defaultURL)
try! FileManager.default.moveItem(at: compactedURL, to: defaultURL)
}
}
http://hadoop.apache.org/docs/r2.1.0-beta/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html
I am try to make the example work well from the above link.but I can't compile the code below
Resource capability = Records.newRecord(Resource.class);
capability.setMemory(512);
amContainer.setResource(capability);
// Set the container launch content into the
// ApplicationSubmissionContext
appContext.setAMContainerSpec(amContainer);
amContainer is ContainerLaunchContext and my hadoop version is 2.1.0-beta.
I did some investigation. I found there's no method "setResource" in ContainerLaunchContext
I have 3 question about this
1) the method has been removed or something?
2) if the method has been removed, how can I do now?
3) is there any doc about yarn, because I found the doc in website is very easy, I hope I can get a manual or something. for example,
capability.setMemory(512);
I don't know it's 512k or 512M according comments in code.
This is actually proper solution to the question. Previous answer might cause incorrect execution !!!
#Dyin I couldn't fit it in the comment ;) Validated for 2.2.0 and 2.3.0
Driver setting up resources for AppMaster:
ApplicationSubmissionContext appContext = app.getApplicationSubmissionContext();
ApplicationId appId = appContext.getApplicationId();
appContext.setApplicationName(this.appName);
// Set up the container launch context for the application master
ContainerLaunchContext amContainer = Records.newRecord(ContainerLaunchContext.class);
Resource capability = Records.newRecord(Resource.class);
capability.setMemory(amMemory);
appContext.setResource(capability);
appContext.setAMContainerSpec(amContainer);
Priority pri = Records.newRecord(Priority.class);
pri.setPriority(amPriority);
appContext.setPriority(pri);
appContext.setQueue(amQueue);
// Submit the application to the applications manager
yarnClient.submitApplication(appContext); // this.yarnClient = YarnClient.createYarnClient();
In ApplicationMaster this is how you should specify resources for containers (workers).
private AMRMClient.ContainerRequest setupContainerAskForRM() {
// setup requirements for hosts
// using * as any host will do for the distributed shell app
// set the priority for the request
Priority pri = Records.newRecord(Priority.class);
pri.setPriority(requestPriority);
// Set up resource type requirements
// For now, only memory is supported so we set memory requirements
Resource capability = Records.newRecord(Resource.class);
capability.setMemory(containerMemory);
AMRMClient.ContainerRequest request = new AMRMClient.ContainerRequest(capability, null, null,
pri);
return request;
}
Some run() or main() method in your AppMaster
AMRMClientAsync.CallbackHandler allocListener = new RMCallbackHandler();
resourceManager = AMRMClientAsync.createAMRMClientAsync(1000, allocListener);
resourceManager.init(conf);
resourceManager.start();
for (int i = 0; i < numTotalContainers; ++i) {
AMRMClient.ContainerRequest containerAsk = setupContainerAskForRM();
resourceManager.addContainerRequest(containerAsk); //
}
Launching containers
You can use the original answer solution (java cmd), but it's just a cherry on top. It should work anyway.
You can set memory available to ApplicationMaster via commend. As such:
// Set the necessary command to execute the application master
Vector<CharSequence> vargs = new Vector<CharSequence>(30);
...
vargs.add("-Xmx" + amMemory + "m"); // notice "m" indicating megabytes, you can use also -Xms combined with -Xmx
... // transform vargs to String commands
amContainer.setCommands(commands);
This should solve your problem. As for the 3 questions. Yarn is rapidly evolving software. My advice forget documentation, get source code and read it. This will answer a lot of your questions.
I recently switched to Spring Framework instead of manually handling JDBC, and it is mostly a good transition. One program started having strange problems, though: if the database is slow, when calling getJdbcTemplate().update( ... ) it sometimes never returns.
After researching a little bit, I switched from Apache DBCP to C3PO, but the problem still came back.
Here's the code I'm using:
public class MyDao extends SimpleJdbcDaoSupport {
private static Logger logger = Logger.getLogger(MyDao.class);
public MyDao(Config config) {
super();
ComboPooledDataSource cpds = new ComboPooledDataSource();
try {
cpds.setDriverClass("com.mysql.jdbc.Driver");
} catch (PropertyVetoException e) {
throw new RuntimeException(e);
}
cpds.setUser("username");
cpds.setPassword("password");
cpds.setJdbcUrl("jdbc:mysql://localhost/schema" +
"?useUnicode=true&characterEncoding=UTF-8");
cpds.setMaxStatements( 180 );
cpds.setPreferredTestQuery("SELECT 1");
cpds.setTestConnectionOnCheckout(true);
this.setDataSource(cpds);
}
public void addToWorkQueue(String item) {
long[] ids = Utils.getItemIds(item);
try {
logger.debug("About to insert to work table");
getJdbcTemplate().update(
"INSERT IGNORE INTO work " +
"SELECT * FROM queue WHERE id_1 = ? AND id_2 = ?",
new Object[] { ids[0], ids[1] }
);
} finally {
logger.debug("Updated work table");
}
}
}
Here's what it looks like in the log file:
2009-07-29 17:37:13.570 com.mycomp.MyDao About to insert into work table
2009-07-29 17:37:13.570 com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool Testing PooledConnection [com.mchange.v2.c3p0.impl.NewPooledConnection#170984c] on CHECKOUT.
2009-07-29 17:37:13.571 com.mchange.v2.c3p0.stmt.GooGooStatementCache checkinAll(): com.mchange.v2.c3p0.stmt.GlobalMaxOnlyStatementCache stats -- total size: 1; checked out: 0; num connections: 1; num keys: 1
2009-07-29 17:37:13.571 com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool Test of PooledConnection [com.mchange.v2.c3p0.impl.NewPooledConnection#170984c] on CHECKOUT has SUCCEEDED.
2009-07-29 17:37:13.571 com.mchange.v2.resourcepool.BasicResourcePool trace com.mchange.v2.resourcepool.BasicResourcePool#d402dd [managed: 3, unused: 2, excluded: 0] (e.g. com.mchange.v2.c3p0.impl.NewPooledConnection#170984c)
2009-07-29 17:37:13.571 com.mchange.v2.c3p0.stmt.GooGooStatementCache com.mchange.v2.c3p0.stmt.GlobalMaxOnlyStatementCache ----> CACHE HIT
2009-07-29 17:37:13.571 com.mchange.v2.c3p0.stmt.GooGooStatementCache checkoutStatement: com.mchange.v2.c3p0.stmt.GlobalMaxOnlyStatementCache stats -- total size: 1; checked out: 1; num connections: 1; num keys: 1
This is where the code hangs. Usually it just goes on like this:
2009-07-29 17:37:13.762 com.mchange.v2.c3p0.stmt.GooGooStatementCache checkinStatement(): com.mchange.v2.c3p0.stmt.GlobalMaxOnlyStatementCache stats -- total size: 1; checked out: 0; num connections: 1; num keys: 1
2009-07-29 17:37:13.763 com.mchange.v2.c3p0.stmt.GooGooStatementCache checkinAll(): com.mchange.v2.c3p0.stmt.GlobalMaxOnlyStatementCache stats -- total size: 1; checked out: 0; num connections: 1; num keys: 1
2009-07-29 17:37:13.763 com.mchange.v2.resourcepool.BasicResourcePool trace com.mchange.v2.resourcepool.BasicResourcePool#d402dd [managed: 3, unused: 2, excluded: 0] (e.g. com.mchange.v2.c3p0.impl.NewPooledConnection#170984c)
2009-07-29 17:37:13.763 com.mycomp.MyDao Updated work table
I don't know why I'm not getting any log message from Spring Framework itself. I added these lines in my main code:
Logger springLogger = Logger.getLogger("org.springframework");
springLogger.setLevel(Level.TRACE);
springLogger.debug("testing spring logger");
The test message shows, but nothing else. Sorry for diverging.
I did notice a slowdown before the hang. The last time the query ran successfully it took a minute and a half to finish, instead of the usual 200ms. The next time, I let it run for 25 minutes before killing the process.
I know I have some problems with my database (InnoDB), which I'm working on, but this seems like after a timeout, Spring Framework just "gives up" and hangs.
Any advice would be appreciated.
Eventually, the problem was avoided by fixing the underlying DB issue.
I was using an InnoDB table as a work queue, which meant I added and removed a whole lot of items to it. The table never had too many rows at a given time, but apparently InnoDB can't handle this sort of work, or as my DBA friend put it, "deleting rows from a table doesn't do anything for performance".
After switching to a much crazier db strategy, which involved creating and dropping tables all the time, the performance was much improved and the hangs went away.
So I guess what I'm saying is, skaffman's comment was probably right. This didn't have anything to do with Spring.